10G-SR interfaces flap when connected to Intel x710 NICs using firmware 6.0.x.
Multiple customers have encountered link connectivity issues when connecting to a server containing an Intel x710 NIC with firmware 6.0.0 or 6.0.1. Intel x710 NICs are common on Dell servers. This issue can be seen on breakout ports as well.
In some cases, the affected interfaces are represented as "ge" interfaces in the hardware. For example, the Broadcom ASIC shows the link as as 10G/GMII:
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discard ops face frame back
ge0( 1) down 10G FD SW No Disable None FA GMII 1518 ge1( 2) down 10G FD SW No Disable None FA GMII 1518
In other cases, the link simply remains as no carrier and can be seen flapping. This is seen upon visual inspection of the Intel x710 NIC; the corresponding link status light will flash green.
- Intel NIC x710 FW: 6.0.0 & 6.0.1
- Broadcom Trident II
- Broadcom Trident II+
- Broadcom Tomahawk
- Mellanox Spectrum
Hardware (server side transceiver):
- Intel AFBR.709DMZ-IN2
There are two workarounds to this issue.
- Downgrade the firmware on the Intel x710 NIC to 5.x.x.
- Using the Intel x710 6.0.x firmware, force the link speed and auto-negotiation settings on the server and on the switch.
The following configuration should be applied to the 10G port on the switch:
auto swpX iface swpX link-speed 10000
This can be done by executing the following NCLU commands:
cumulus@switch:~$ net add interface swpX link speed 10000 cumulus@switch:~$ net pending cumulus@switch:~$ net commit
The following command should be applied to the 10G ports on the server to force it to advertise 10000Mb/s:
ethtool -s ethX advertise 0x80000000000
Note: If neither workaround is effective, contact Cumulus Support or Dell Support.