No Connectivity with Intel x710 NICs on FW 6.0.0

Follow

Issue

10G-SR interfaces flap when connected to Intel x710 NICs using firmware 6.0.x.

Multiple customers have encountered link connectivity issues when connecting to a server containing an Intel x710 NIC with firmware 6.0.0 or 6.0.1. Intel x710 NICs are common on Dell servers. This issue can be seen on breakout ports as well.

In some cases, the affected interfaces are represented as "ge" interfaces in the hardware. For example, the Broadcom ASIC shows the link as as 10G/GMII:

           ena/ speed/   link auto STP                 lrn inter max loop

port      link duplex    scan neg? state  pause discard ops face  frame back

 ge0(  1) down   10G FD SW   No Disable      None FA GMII 1518      
 ge1(  2) down   10G FD SW   No Disable      None FA GMII 1518

In other cases, the link simply remains as no carrier and can be seen flapping. This is seen upon visual inspection of the Intel x710 NIC; the corresponding link status light will flash green.

Environment

Firmware:

  • Intel NIC x710 FW: 6.0.0 & 6.0.1

Hardware (ASIC):

  • Broadcom Trident II
  • Broadcom Trident II+
  • Broadcom Tomahawk
  • Mellanox Spectrum

Hardware (server side transceiver):

  • Intel AFBR.709DMZ-IN2

Workaround

There are two workarounds to this issue.

  • Downgrade the firmware on the Intel x710 NIC to 5.x.x.
  • Using the Intel x710 6.0.x firmware, force the link speed and auto-negotiation settings on the server and on the switch.

    The following configuration should be applied to the 10G port on the switch:
auto swpX
iface swpX
   link-speed 10000

This can be done by executing the following NCLU commands:

cumulus@switch:~$ net add interface swpX link speed 10000
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The following command should be applied to the 10G ports on the server to force it to advertise 10000Mb/s:

ethtool -s ethX advertise 0x80000000000

Note: If neither workaround is effective, contact Cumulus Support or Dell Support.

Have more questions? Submit a request

Comments

Powered by Zendesk