The Intel FTLX8571D3BCV-IT SFP and the equivalent Finisar SFP (FTLX8571D3BCV-IT) do not come up on an Edge-Core AS5712-54X switch. Unexpected ethtool output was noted while troubleshooting the issue.

This issue is fixed in Cumulus Linux 2.5.10.

An error like the following has been seen while installing Cumulus Linux using cl-img-install:

stderr: /usr/cumulus/bin/cl-img-install: 64: /usr/cumulus/bin/cl-img-install: arithmetic expression: expecting primary: " (  % 2 ) + 1 "

Sometimes this issue appears if you power off the switch while installing Cumulus Linux into slot 2.

This issue is fixed in Cumulus Linux 2.5.10.

When receiving a route from an IPv6 eBGP neighbor, Cumulus Linux always installs the link local address instead of the peer address or the global address, even with a route map. This can cause issues in circumstances like when the neighbor router is advertising a virtual IP address as the next hop for a load balancer, whereas the next hop must be the global address, not a link local address.

This issue is fixed in Cumulus Linux 2.5.10 — now the link local address no longer appears in the BGP table entry and the next hop is set to the globally-defined address.

If you remove the link-speed and link-duplex configurations from one or more interfaces, the settings for these configurations do not return to their defaults.

The workaround involved re-configuring the port by explicitly configuring link-speed and link-duplex under the iface stanza.

This issue is fixed in Cumulus Linux 2.5.10.

Sometimes the BGP service, bgpd, crashes when an assert fires.

This issue is fixed in Cumulus Linux 2.5.10.

To configure the dhcrelay service to forward IPv4 (only) DHCP packets to a server and ensure that the source IP address of the relayed packet is the same as the gateway IP address, edit the /etc/default/isc-dhcp-relay file and adding OPTIONS=--giaddr-src, setting it to the gateway IP address, then restart the dhcrelay service.

cumulus@leaf:~$ sudo service dhcrelay restart

This issue is fixed in Cumulus Linux 2.5.10.

A memory leak occurred during the blackhole route addition. switchd did not get the callback, though the routes had been added by Quagga. During deletion, switchd received an add callback for the routes as well as a delete callback for the same routes, and the routes were deleted from the cache.

This issue is fixed in Cumulus Linux 2.5.10.

This issue is fixed in Cumulus Linux 2.5.10.

An error occurred when the dependency graph was generated for the last saved interfaces file. If the saved config already contained bridge interfaces and dependencies, which resulted in the bridge dependency information being skipped, and not included in the dependency graph. The dependency generation process has been corrected to ensure that the information is included in the graph, even if dependences are already present.

This issue is fixed in Cumulus Linux 2.5.10.

In some environments, multiple routes present in the BGP received-routes list and BGP table were not present within zebra or the kernel. To resolve this issue, iBGP peerings are now brought down upon link down, if they are tracked by single-hop BFD peering. This ensures many of the interim state transitions that caused the issues no longer take place.

Note: iBGP BFD sessions are treated as multi-hop by default in 2.5.x. The session will need to be explicitly configured as single hop.

This issue is fixed in Cumulus Linux 2.5.10.

During Multipath route selection, it is possible that the old multipath list of routes for a destination may include routes from peers that are no longer Established, when multiple peers are flapping. When the new multipath list is compared against the old list to identify changes, additional checks were needed to avoid comparing connection addresses if the peer was not Established. Those checks have now been included in version 2.5.10.

On Edge-Core 56100 switches, the SFP/cable did not work when connected to swp41. This was caused because the bit range for swp41 to swp48 was configured incorrectly.

The range has been corrected, and the issue is fixed in Cumulus Linux 2.5.10.

An update to /usr/lib/cumulus/datapath-update was backported to correct a tuning issue regarding the shared buffer limit values. 

swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
 inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:4231 errors:0 dropped:0 overruns:0 frame:0
 TX packets:4342 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:412115 (402.4 KiB) TX bytes:425688 (415.7 KiB)

cumulus@switch$ sudo ifdown swp1
cumulus@switch$ sudo ifconfig swp1
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
 BROADCAST MULTICAST MTU:1500 Metric:1
 RX packets:4248 errors:0 dropped:0 overruns:0 frame:0
 TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB)

cumulus@switch$ sudo ifconfig swp1
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
 BROADCAST MULTICAST MTU:1500 Metric:1
 RX packets:4248 errors:0 dropped:0 overruns:0 frame:0
 TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB)
 cumulus@switch$ sudo ifup swp1
 ADDRCONF(NETDEV_UP): swp1: link is not ready
 cumulus@switch$ sudo ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: /
     link becomes ready 
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 
 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
 inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:4250 errors:0 dropped:0 overruns:0 frame:0
 TX packets:4362 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:414178 (404.4 KiB) TX bytes:427610 (417.5 KiB)

cumulus@switch$ 

cumulus@switch:~$ sudo ifconfig swp1
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link 
 inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:98 errors:0 dropped:0 overruns:0 frame:0
 TX packets:111 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500 
 RX bytes:13310 (12.9 KiB) TX bytes:12786 (12.4 KiB)

cumulus@switch$ sudo ifconfig swp1 down
cumulus@switch$ sudo ifconfig swp1
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
 BROADCAST MULTICAST MTU:1500 Metric:1
 RX packets:126 errors:0 dropped:0 overruns:0 frame:0
 TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:16998 (16.5 KiB) TX bytes:15998 (15.6 KiB)

cumulus@switch$ sudo ifconfig swp1 up
 ADDRCONF(NETDEV_UP): swp1: link is not ready

cumulus@switch$ sudo ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: 
    link becomes ready 
 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 
 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 
 RX packets:130 errors:0 dropped:0 overruns:0 frame:0
 TX packets:149 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:17474 (17.0 KiB) TX bytes:17154 (16.7 KiB) 

Ports can be aggregated into a larger interface in Cumulus Linux. Unfortunately support for aggregated ports is not yet supported when running cl-phy-update.

If there are any ganged ports during a software upgrade, it is recommended to ungang these ports

If either of the following is configured:

net.ipv4.ip_forward == 0 

or:

net.ipv6.conf.all.forwarding == 0 

The hardware still forwards packets if there is a neighbor table entry pointing to the destination.

In certain conditions, Quagga's bgpd service silently closes the peering without sending a notification. For example, if BGP receives a message with an invalid message type or invalid message length.

Ideally on any one of these cases, bgpd should send out a notification message to the peer.

General functionality of BGP4 is not affected.

router bgp 65000
 bgp router-id 0.0.0.4 
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 bgp cluster-id 0.0.0.4 
 bgp bestpath as-path multipath-relax 
 redistribute connected 
 neighbor 14.0.0.1 remote-as 65000 
 neighbor 14.0.0.1 route-reflector-client 
 neighbor 14.0.0.1 activate 
 neighbor 14.0.0.1 next-hop-self 
 neighbor 14.0.0.9 remote-as 65000 
 neighbor 14.0.0.9 activate 
 neighbor 14.0.0.9 next-hop-self 
 neighbor 2001:ded:beef::1 remote-as 65000 
 neighbor 2001:ded:beef:2::1 remote-as 65000 
 maximum-paths 4 
 maximum-paths ibgp 4 
 ! 
 address-family ipv6 
 redistribute connected 
 neighbor 2001:ded:beef::1 activate 
 neighbor 2001:ded:beef::1 next-hop-self 
 neighbor 2001:ded:beef:2::1 route-reflector-client 
 neighbor 2001:ded:beef:2::1 activate 
 neighbor 2001:ded:beef:2::1 next-hop-self 
 maximum-paths 4 
 maximum-paths ibgp 4 
 exit-address-family 

cumulus@switch:$ show ip bgp neighbor 14.0.0.1 
 BGP neighbor is 14.0.0.1, remote AS 65000, local AS 65000, internal link
 BGP version 4, remote router ID 0.0.0.6 
 BGP state = Established, up for 00:23:49
 Last read 23:31:36, hold time is 180, keepalive interval is 60 seconds 
 Neighbor capabilities: 
 4 Byte AS: advertised and received 
 Route refresh: advertised and received(old & new)
 Address family IPv4 Unicast: advertised and received 
 Message statistics: 
 Inq depth is 0 
 Outq depth is 0
 Sent Rcvd 
 Opens: 2 0
 Notifications: 0 0
 Updates: 1 1
 Keepalives: 25 24 
 Route Refresh: 0 0
 Capability: 0 0 
 Total: 28 25 
 Minimum time between advertisement runs is 5 seconds
 For address family: IPv4 Unicast 
 >>>>>>>>>>>>>>>>>>>>>> ROUTE REFLECTOR CLIENT NOT DISPLAYED 
 NEXT_HOP is always this router 
 Community attribute sent to this neighbor(both) 
 6 accepted prefixes 
 Connections established 1; dropped 0 
 Last reset never 
 Local host: 14.0.0.2, Local port: 179 
 Foreign host: 14.0.0.1, Foreign port: 40290 
 Nexthop: 14.0.0.2 
 Nexthop global: 2001:ded:beef::2 
 Nexthop local: fe80::202:ff:fe00:4
 BGP connection: non shared network
 Read thread: on Write thread: off 
 cumulus@switch:$ 

Define in following order 
 address-family ipv4 unicast
 neighbor 14.0.0.9 activate 
 neighbor 14.0.0.9 next-hop-self
 neighbor 14.0.0.9 route-reflector-client >>> Must be after Activate 
 exit-address-family 
 neighbor 2001:ded:beef:2::1 remote-as 65000
 address-family ipv6 unicast 
 redistribute connected
 maximum-paths 4 
 maximum-paths ibgp 4 
 neighbor 2001:ded:beef:2::1 activate 
 neighbor 2001:ded:beef:2::1 next-hop-self 
 neighbor 2001:ded:beef:2::1 route-reflector-client >>> Must be after activate 
 exit-address-family 
 Runtime status after change: 

cumulus@switch:$ show ip bgp neighbors 14.0.0.9 
 BGP neighbor is 14.0.0.9, remote AS 65000, local AS 65000, internal link 
 BGP version 4, remote router ID 0.0.0.7 
 BGP state = Established, up for 00:13:59
 Last read 22:35:13, hold time is 180, keepalive interval is 60 seconds 
 Neighbor capabilities: 
 4 Byte AS: advertised and received 
 Route refresh: advertised and received(old & new) 
 Address family IPv4 Unicast: advertised and received 
 Message statistics: 
 Inq depth is 0 
 Outq depth is 0
 Sent Rcvd 
 Opens: 1 1
 Notifications: 0 0 
 Updates: 2 1 
 Keepalives: 15 14
 Route Refresh: 0 0
 Capability: 0 0 
 Total: 18 16 
 Minimum time between advertisement runs is 5 seconds
 For address family: IPv4 Unicast 
 Route-Reflector Client >>>>>>>>>> PLEASE NOTE ME 
 NEXT_HOP is always this router 
 Community attribute sent to this neighbor(both) 
 6 accepted prefixes 
 Connections established 1; dropped 0 
 Last reset never 
 Local host: 14.0.0.10, Local port: 38813 
 Foreign host: 14.0.0.9, Foreign port: 179
 Nexthop: 14.0.0.10 
 Nexthop global: 2001:ded:beef:2::2 
 Nexthop local: fe80::202:ff:fe00:6 
 BGP connection: non shared network 
 Read thread: on Write thread: off 
 cumulus@switch:$

routes: 16384 <<<< if all routes are ipv4 
 long mask routes 256 <<<< i.e., routes with a mask longer 
       than the route mask limit 
 route mask limit 64
 host_routes: 8192 
 ecmp_nhs: 4044 
 ecmp_nhs_per_route: 52 

cumulus@switch:~$ sudo cl-resource-query
 hosts : 3 
 all routes : 29 
 IP4 routes : 17 
 IP6 routes : 12 
 nexthops : 3 
 ecmp_groups : 0
 ecmp_nexthops : 0
 mac entries : 0 / 131072 
 bpdu entries : 500 / 512

Cumulus Linux does not provide SNMP support for Quagga.

You can get this information via Nagios. 

However, it's possible to get it via SNMP by:

• Writing a pass persist script in Perl or Python by filling in the OSPF or BGP (rfc) MIBs manually
• Creating your own private MIB for the information you need.

cumulus@switch:$ sudo vtysh -c 'show int swp8' 
Interface swp8 is up, line protocol is up 
PTM status: fail
index 10 metric 1 mtu 1500 
 flags: <UP,BROADCAST,RUNNING,MULTICAST>
 HWaddr: 44:38:39:00:03:88 
 inet 12.0.0.225/30 broadcast 12.0.0.227 
 inet6 2001:cafe:0:38::1/64 
 inet6 fe80::4638:39ff:fe00:388/64 
cumulus@switch:$ ip addr show | grep swp8 
 10: swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc pfifo_fast state UP qlen 500 
  inet 12.0.0.225/30 brd 12.0.0.227 scope global swp8 
 104: swp8.2049@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.229/30 brd 12.0.0.231 scope global swp8.2049 
 105: swp8.2050@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.233/30 brd 12.0.0.235 scope global swp8.2050 
 106: swp8.2051@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.237/30 brd 12.0.0.239 scope global swp8.2051 
 107: swp8.2052@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.241/30 brd 12.0.0.243 scope global swp8.2052 
 108: swp8.2053@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP>
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.245/30 brd 12.0.0.247 scope global swp8.2053 
 109: swp8.2054@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP> 
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.249/30 brd 12.0.0.251 scope global swp8.2054
 110: swp8.2055@swp8: <BROADCAST,MULTICAST,UP,LOWER_UP>
  mtu 1500 qdisc noqueue state UP 
  inet 12.0.0.253/30 brd 12.0.0.255 scope global swp8.2055
cumulus@switch:$ bgp sessions: 
 12.0.0.226 ,4 ,64057 , 958 , 1036 , 0 , 0 , 0 ,15:55:42, 0, 10472 
 12.0.0.230 ,4 ,64058 , 958 , 1016 , 0 , 0 , 0 ,15:55:46, 187, 10285
 12.0.0.234 ,4 ,64059 , 958 , 1049 , 0 , 0 , 0 ,15:55:40, 187, 10285 
 12.0.0.238 ,4 ,64060 , 958 , 1039 , 0 , 0 , 0 ,15:55:45, 187, 10285 
 12.0.0.242 ,4 ,64061 , 958 , 1014 , 0 , 0 , 0 ,15:55:46, 187, 10285 
 12.0.0.246 ,4 ,64062 , 958 , 1016 , 0 , 0 , 0 ,15:55:46, 187, 10285 
 12.0.0.250 ,4 ,64063 , 958 , 1029 , 0 , 0 , 0 ,15:55:43, 187, 10285 
 12.0.0.254 ,4 ,64064 , 958 , 1036 , 0 , 0 , 0 ,15:55:44, 187, 10285 

Before running any apt-get commands or after changing the source.list file in /etc/apt, you need to run apt-get update.

Device names, including interface names, in Cumulus Linux cannot exceed 16 characters – including the terminator. Cumulus Linux truncates longer interface names.

To avoid this issue, do not assign long names to your interfaces.

The following example configuration reproduces this issue:

cumulus@switch:/sys/class/net$ grep 'iface br' /etc/network/interfaces 
iface br2-pubmgmt inet static
iface br3-prvmgmt inet manual
iface br400-quarantine inet manual
iface br401-peering-1k5 inet manual
iface br402-peering-9k inet manual
iface br500-pi-exa inet manual
iface br501-akamai-exa inet manual
iface br502-exa-internetfactory inet manual
cumulus@switch:/sys/class/net$ brctl show | grep br
bridge name	bridge id	 STP enabled	interfaces
br2-pubmgmt	 8000.089e01cebe37	no	 bond0.2
br3-prvmgmt	 8000.089e01cebe3a	no	 bond0.3
br400-quarantin	 8000.089e01cebe37	no	 bond0.400
br401-peering-1	 8000.089e01cebe3a	no	 bond0.401 <<<

The Cumulus Linux 2.2 repository contains two versions of Chef, the automation tool: 11.6.2 (the current version) and 10.30.4.

To install the latest version, connect to the switch and use apt-get:

cumulus@switch:~$ sudo  apt-get install chef

To install 10.30.4, connect to the switch and use apt-get:

cumulus@switch:~$ sudo apt-get install chef=10.30.4-0.debian.7.3 

In a VXLAN using VMware NSX, ovsdb-server cannot select the loopback interface as the source IP address. This causes TOR registration to the controller to fail.

To work around this issue, run:

cl-bgp redistribute add connected

10GTek 10G SR cables exhibit high rate of errors on Penguin Arctica 4804X switch

Some PHY-less Penguin Arctica 4804X platforms using 10GTek 10G MM SR cables exhibit high rates of errors and low bandwidth one direction.

Link not coming up (NO-CARRIER) on Penguin Arctica 4804xp with CAB-10GSFP-P9M 10Gtek 9 meter cable

The link does not come up on a Penguin Arctica 4804xp 720G PHY-less switch with 10Gtek 9 meter cables (CAB-10GSFP-P9M). You can determine this by running:

cumulus@switch:~$ ip link show swp15
17: swp15: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc 
    pfifo_fast state DOWN mode DEFAULT qlen 500
    link/ether 44:38:39:00:70:a1 brd ff:ff:ff:ff:ff:ff

Cumulus Linux supports copper passive cables no longer than 7m.

PTM may crash when a large topology file has a syntax error

If there is a syntax error in a large (~ 4000 entries) topology.dot file, PTM may crash while reading/parsing this file. This can occur because the libcgraph API that PTM uses for parsing the file cannot handle the error. You should check topology.dot for syntax errors using a Graphviz tool like dot or dotty that can identify the syntax error.

As a result, the TOR registration to the controller fails.

To work around this issue, run:

cl-bgp redistribute add connected

It's been observed that port LEDs behave differently depending upon the make and model of the switch. For example:

• Agema AG-7448CU: the LED is off when the link is up. It blinks on briefly when there is traffic.
• Edge-Core AS4600-54T: the LED is off when the link is up. It blinks on briefly when there is traffic.
• QuantaMesh T3048-LY2R: the LED is on when the link is up. It blinks off briefly when there is traffic.

Cumulus Networks is currently working to fix this issue.

Cumulus Linux triggers a route-map update before the user finishes editing the route map, resulting in an incorrect route map being used. The route-map update trigger should only occur when user finishes editing the map.

Cumulus Networks is working to fix this issue.

Given the following conditions:

• You have not configured a loopback anycast IP address in /etc/network/interfaces
• You enabled the vxsnd (service node daemon) log to automatically add anycast IP addresses

When you restart networking (with service networking restart), the anycast IP address gets removed from the loopback interface.

To prevent this issue from occurring, you should specify an anycast IP address for the loopback interface in both /etc/network/interfaces and vxsnd.conf. This way, in case the vxsnd fails, you can withdraw the IP address.

ifupdown2 doesn't bring up loopback interface (lo) by default.

One effect is that since monit uses loopback interfaces, if you delete the lo interface in /etc/network/interfaces, monit commands will not work.

Even though bond-min-links defaults to 0, Cumulus Networks recommends a minimum link setting of 1 or higher for bonds. Cumulus Linux issues a warning when the setting is 0 or not specified.

For QuantaMesh BMS T3048-LY9 switches, the 10GTek QSFP-SR4 optical cable is not supported in Cumulus Linux 2.5.10.

Use the JDSU JQP-04SWAA1 or comparable Multi-Mode QSFP+ cable instead.

Cumulus Networks is working to support this transceiver in a future release.

Under rare and unusual conditions on various x86 Trident II platforms (such as the Penguin Computing Arctica 4806XP, switchd does not start after system reboot and displays this error message:

MiimTimeOut:soc_miim_write, timeout (id=0x1da addr=0x1f data=0x8630)
system_init: Device reset failed: Operation timed out
PORT: Error: bcm ports not initialized
Error: file /etc/bcm.d/rc.ports_0: line 42 (error code -1): script terminated
ERROR loading rc script on unit 0
loading of rc script failed, aborting!

This condition is extremely rare, and does not occur if the switch booted correctly and is operational.

The bridge fdb add command handles the IP address type in a non-standard way:

• bridge fdb add [MAC address] dev [INTERFACE] master defaults to a static address
• bridge fdb add [MAC address] dev [INTERFACE] master temp maps to dynamic address
• bridge fdb add [MAC address] dev [INTERFACE] master local maps to permanent address

inotify is not supported by the overlayfs root filesystem on PowerPC platforms.

As a best practice for when multiple next-hops are going to the same switch, you should configure a bond with an uplink port.

The path that Cumulus Linux originates should not be invalid since there is a matching route in the RIB. The import check works fine for IPv4 routes.

Cumulus Linux does not support STP on bonds in LACP bypass all active mode. To avoid this issue, enable STP BPDU guard for all bonds in LACP bypass all-active mode. For information, read this document.

If you try to remove the default route from eth0 (either by commenting out or removing the gateway statement in the eth0 configuration), the route remains after running ifup.

To work around this issue, first run ifdown, then ifup on the interface via the console. After this, the route disappears.

Other 1G switches support 32K (32768) entries. This is a known issue that should be fixed in a future release of Cumulus Linux.

If you upgraded from a Cumulus Linux version earlier than 2.5.3 using cl-img-install, some user IDs in /etc/passwd are different than they were under a previous version. This issue can occur if you copy /etc/passwd and /etc/group from the active partition after you upgrade using cl-img-install. Overwriting these files results in LLDP failures as you can see here:

cumulus@switch:~ $ sudo lldpcli show  neighbors 
-------------------------------------------------------------------------------
LLDP neighbors:
2015-06-08T20:11:34 [WARN/control] unable to connect to socket 
    /var/run/lldpd.socket: Permission denied
2015-06-08T20:11:34 [WARN/lldpctl] not able to get the list of 
    interfaces. Unable to connect to lldpd daemon
-------------------------------------------------------------------------------

Note: If you upgrade using apt-get dist-upgrade, the passwd file doesn't change from the earlier version of Cumulus Linux you had installed. 

To work around this issue, after you overwrite the passwd file, change the owner of the lldpcli file, then restart lldpd.

cumulus@switch:~ $ sudo chown _lldpd /usr/sbin/lldpcli
cumulus@switch:~ $ 
cumulus@switch:~ $ sudo service lldpd restart
[ ok ] Restarting LLDP daemon: lldpd.
cumulus@switch:~ $ 

Another side effect of this issue is that the owner for the Quagga log file, /var/log/quagga, changed. Instead of quagga/quagga, it becomes snmp/ntp. To fix this issue, change the owner of the log file:

cumulus@switch:~$ sudo chown quagga:quagga /var/log/quagga

Issuing a poweroff or shutdown command with certain options shuts down the switch, but it cannot be powered on again. The options that cause this issue are:

• shutdown -h, shutdown -P
• poweroff -n, poweroff -d, poweroff -f, poweroff -i, poweroff -h, poweroff -p
• halt -n, halt -d, halt -f, halt -i, halt -h, halt -p
• init 0

Issuing a reboot command, or using other options, does not trigger this issue.

This issue affects only Dell S4048-ON switches with BIOS version 3.21.0.2 or earlier. To determine the BIOS version of the switch, run:

cumulus@switch:~$ sudo dmidecode -s system-version
3.21.0.2

This is a known issue, and Dell is issuing a fix soon.

When you enable management VRF, the clagd-backup-ip setting does not work over the switch ports.

To work around this issue, use a backup IP address that is reachable over eth0.

With cut-though mode enabled and link pause is asserted, Cumulus Linux generates an TOVR and TUFL ERROR; certain error counters increment on given physical port.

cumulus@switch:~$ sudo ethtool -S swp49 | grep Error
HwIfInDot3LengthErrors: 0
HwIfInErrors: 0
HwIfInDot3FrameErrors: 0
SoftInErrors: 0
SoftInFrameErrors: 0
HwIfOutErrors: 35495749
SoftOutErrors: 0

cumulus@switch:~$ sudo ethtool -S swp50 | grep Error
HwIfInDot3LengthErrors: 3038098
HwIfInErrors: 297595762
HwIfInDot3FrameErrors: 293710518

To work around this issue, disable link pause or disable cut-through in /etc/cumulus/datapath/traffic.conf.

To disable link pause, comment out the link_pause* section in /etc/cumulus/datapath/traffic.conf:

cumulus@switch:~$ sudo nano /etc/cumulus/datapath/traffic.conf 
#link_pause.port_group_list = [port_group_0]
#link_pause.port_group_0.port_set = swp45-swp54
#link_pause.port_group_0.rx_enable = true
#link_pause.port_group_0.tx_enable = true

To enable store and forward switching, set cut_through_enable to false in /etc/cumulus/datapath/traffic.conf:

cumulus@switch:~$ sudo nano /etc/cumulus/datapath/traffic.conf 
cut_through_enable = false

The Hurricane2 chipset used on Cumulus RMP switches does not support ACL rules that specify ICMP type or code. These rules are ignored and are not installed in hardware without providing any feedback to the user.

Cumulus Networks is aware of this issue but there is no known workaround at this time.

To work around this issue, remove any old redistribute command configurations before adding a new one with or without route-map as a parameter.

For example, if OSPF has a redistribute configuration such as redistribute bgp route-map redist-map-name, you would enable redistribution without a route-map by following these steps in OSPF configuration mode:

1. no redistribute bgp
2. redistribute bgp

You would perform a similar sequence of commands for redistribution changes in BGP as well.

When performing a fast reboot (using the reboot -f command), the switch ports remain up and only the control plane goes down.

You should only use this command under abnormal circumstances, because it bypasses normal shutdown.

Link may fail to come up between Edge-Core AS6701-32X switches and Intel Fortville XL710-based NICs when used with 40G DACs. 

To correct this issue, upgrade to the most recent Intel i40e NIC driver (greater than version 1.3.49) and firmware (greater than 5.02 0x80002281 0.0.0). The updated NIC drivers may be available through the Linux distribution installed on the server, or alternatively are available from Intel directly.

Running out of disk space can render a switch non-functional if the file system becomes full. This issue particularly affects PowerPC switches with small disk drives (1GB of capacity or less), which include the following:

• Edge-Core AS4600-54T
• Penguin Computing Arctica 3200XL, Arctica 4804X, Arctica 4804I

One action that can consume a lot of disk space is upgrading Cumulus Linux. Before you upgrade, you should determine how much free space is on your switch. Run df -m and make sure /mnt/root-rw (the read-write partition of the overlay file system) has at least 200MB of free disk space. The sample output below shows 737MB free for this partition:

cumulus@switch:~$ df -m
Filesystem     1M-blocks  Used Available Use% Mounted on
dev                   10     0        10   0% /dev
/dev/sysroot1         96    96         0 100% /mnt/root-ro
/dev/mmcblk0p3      1212   414       737  36% /mnt/root-rw
overlayfs:root      1212   414       737  36% /
tmpfs                 10     1        10   1% /run
tmpfs                 10     0        10   0% /run/lock
tmpfs                100     0       100   0% /tmp
tmpfs                100     1       100   1% /var/tmp
tmpfs                980     0       980   0% /run/shm
initramfs            980     3       978   1% /mnt/initramfs
/dev/persist         119     2       111   2% /mnt/persist

If your hard disk has less than 200MB and you want to upgrade Cumulus Linux using apt-get dist-upgrade, please contact customer support.

Attempting to install an IPv6 default route on eth0 with a source address fails at reboot or when running ifup on eth0. 

The first execution of  ifup -dv returns this warning and does not install the route:

cumulus@switch:~$ sudo ifup -dv eth0
warning: eth0: post-up cmd '/sbin/ip route add default via 2001:620:5ca1:160::1 /
src 2001:620:5ca1:160::45 dev eth0' failed (RTNETLINK answers: Invalid argument)<<<<<<<<<<

Running ifup a second time on eth0 successfully installs the route. 

There are two ways you can work around this issue. 

1. Add a sleep 2 to the eth0 stanza in /etc/network/interfaces:

   iface eth0 inet6 static
       address 2001:620:5ca1:160::45/64
       post-up /bin/sleep 2s
       post-up /sbin/ip route add default via 2001:620:5ca1:160::1 src 2001:620:5ca11
   :160::45 dev eth0
   

2. Exclude the src parameter to the ip route add that causes the need for the delay. If the src parameter is removed, the route is added correctly.

   iface eth0 inet6 static
       address 2001:620:5ca1:160::45/64
      post-up /sbin/ip route add default via 2001:620:5ca1:160::1 dev eth0
   
   cumulus@switch:~$ ifdown eth0
   Stopping NTP server: ntpd.
   Starting NTP server: ntpd.
   cumulus@switch:~$ ip -6 r s
   cumulus@switch:~$ ifup eth0
   Stopping NTP server: ntpd.
   Starting NTP server: ntpd.
   cumulus@switch:~$ ip -6 r s
   2001:620:5ca1:160::/64 dev eth0  proto kernel  metric 256 
   fe80::/64 dev eth0  proto kernel  metric 256 
   default via 2001:620:5ca1:160::1 dev eth0  metric 1024 
   cumulus@switch:~$

Cumulus Linux does not support the purging existing addresses on interfaces with multiple iface stanzas. Doing so can result in the configuration of multiple addresses for an interface after you change an interface address and reload the configuration with ifreload -a. If this happens, you must shut down and restart the interface with ifup and if down, or manually delete superfluous addresses with ip address delete specify.ip.address.here/mask dev DEVICE.

This behavior is documented in the Cumulus Linux user guide.

Starting with Cumulus Linux 2.5.4, the switchd and ptmd log files now use syslog instead of the Cumulus Linux log library. The log file names have not changed. Other changes include:

• High precision timestamps have been enabled for all rsyslog log files. These changes only have a direct effect on customers who parse the log file with scripts or other programs, because the timestamp format has changed. Here is an example of the new format:
  

  2015-08-14T18:21:43.337804+00:00 cumulus switchd[3629]: 
   switchd.c:1409 switchd version 1.0-cl2.5+5

• Cumulus Linux logging rules have been moved from /etc/rsyslog.conf to files in /etc/rsyslog.d
• logrotate rules in /etc/logrotate.d have been updated to signal rsyslog instead of switchd. 
• switchd.log now has timestamps for all lines logged, whereas the log library format did not have timestamps for messages with embedded newlines, such as some of the startup messages. You can change the switchd logging to use the log library; see man switchd for details.

The LNV packages are not installed when you upgrade Cumulus Linux. You can get the latest version of LNV for this release of Cumulus Linux in one of two ways:

• Do a full binary image install of Cumulus Linux, using cl-img-install
• Install the LNV packages for the registration and service node daemons using apt-get install vxfld-vxrd and/or apt-get install vxfld-vxsnd, depending upon how you intend to use LNV

In Cumulus Linux 2.5.4 and earlier, if an SFP+ is plugged into a QSFP-to-SFP adapter (such as the Mellanox MAM1Q00A-QSA), then DOM data from SFP+ was not available.

Starting with Cumulus Linux 2.5.5, DOM data for an SFP+ plugged into a QSFP-to-SFP adapter is now available, however, not by default. In order to see the DOM data using ethtool, enable the sfp_combatible sysfs node for that specific port:

1. Identify the EEPROM device for that port:

   cumulus@switch:~$ cat /var/lib/cumulus/sfptab | grep swp51s0
   swp51s0	qsfp+	dts-sysfs	/sys/class/eeprom_dev/eeprom53/device/eeprom
   cumulus@switch:~$

2. Set sfp_compatible to 1 for that device:

   cumulus@switch:~$ sudo echo 1 > /sys/class/eeprom_dev/eeprom53/device/sfp_compatible
   cumulus@switch:~$ cat /sys/class/eeprom_dev/eeprom53/device/sfp_compatible
   1
   cumulus@switch:~$

3. DOM data for the SFP+ plugged into the cage is accessible using ethtool:

   cumulus@switch:~$ sudo ethtool -m swp51s0

Starting with Cumulus Linux 2.5.5, a change in the vxrd registration message resulted in creating an incompatibility such that messages from vxrd in Cumulus Linux 2.5.5 and later are not understood by vxsnd on Cumulus Linux 2.5.4. This incompatibility has the following impacts:

• In Head End Replication mode, vxsnd cannot build and distribute the vxrd database.
• In Service Node Replication mode, vxsnd must rely on learning about any vxrd by analyzing BUM packets.
• The reverse combination works fine (the vxsnd is running on Cumulus Linux 2.5.5 or later and the vxrd is running on Cumulus Linux 2.5.4).

To work around this issue, ensure you upgrade the vxsnd from version 2.5.4 to 2.5.10 before you upgrade any vxrd.

While OSPF area ID configuration in Quagga allows for the value to be specified in either dotted decimal format, or as an integer, values specified as an integer will be converted into dotted decimal format when displayed, causing potential confusion for the operator.

This issue does not impact OSPF functionality; only the display output. However, it is recommended that the OSPF area ID is specified in dotted decimal format for consistency.

Deleting a line where a neighbor or peer is associated with a peer group, but the neighbor doesn't have a remote-as specified as part of the peer group specification, causes quagga-reload to fail, as the remote-as is unspecified.

To work around this issue, delete the line that says neighbor x.x.x.x remote-as 0, and replace it with either the correct AS, or associate it with a peer group that has the remote-as specified, then restart Quagga.

DNS is not handled correctly for static IP addresses by the current implementation of management VRF, as it is built around DHCP use cases. While static IP addresses will generally work, rules need to be set to point the entries in resolv.conf so that DNS lookups occur properly. For this reason, DNS lookup is not supported in 2.5.10.

An incorrect result can be returned when using ip route get with management MRF, as it consults the mgmt routing table by default, rather than the main routing table. It is not possible to specify a routing table to look up, or to specify just an iif, or a mark, with ip route get. Instead, to get the desired result, specify either the loopback IP address, or both the loopback IP address and the device, as shown below:

ip route get 10.1.2.2 from 10.0.0.1

The isc-dhcp-server daemon attempts to start by default after upgrading to Cumulus Linux 2.5.10, and rebooting, causing the following error message to display before the login prompt:

[FAIL] startpar: service(s) returned failure: isc-dhcp-server ... failed!

This issue is harmless and has no functionality impact.

If clagd fails at startup time and if you later run ifreload -av -X eth0, the clagd service does not start.

One result is that you cannot monitor clagd with jdoo.

clagd parameters are set through ifupdown2 and stored in /etc/network/interfaces.d/clag. This file is sourced from /etc/network/interfaces.

For details on this issue and how to upgrade, read this article.

The Supermicro X3648S uses a different PSU (DPS-550) than the equivalent Penguin Arctica 4806XP switch (DPS-460). Cumulus Linux was written to work with the DPS-460, but the DPS-550 behaves differently. 

On the Penguin Arctica 4806XP, a PWM value is written to the DPS-460 fan, and the power supply fan remains at that setting. But on the Supermicro X3648S, when a PWM value is written to the DPS-550 fan, the power supply fan initially goes to the setting; however, after five seconds, it may revert back to its internal setting.

As a result, you will see smond warning messages that the fan input RPM is lower than expected, which can be safely ignored.

There are two known issues regarding isc-dhcp-relay:

• If isc-dhcp-relay is already running and a host-facing interface is flapped (brought down then up using ifupdown2), the DHCP relay will not work for that interface until isc-dhcp-relay is restarted.
• If a logical interface defined in /etc/default/isc-dhcp-relay is down when isc-dhcp-relay is started or restarted, then isc-dhcp-relay will not start; thus, the DHCP relay will not work for any interfaces.

To work around the issue, apply the following configuration to each interface specified in INTERFACES from /etc/default/isc-dhcp-relay to the corresponding iface in /etc/network/interfaces:

post-up test -e /var/run/boot.done && service isc-dhcp-relay restart