Overview

These release notes support Cumulus Linux 1.5.3 and describe currently available features and known issues.

Licensing

Cumulus Linux is licensed on a per-instance basis. Each network system is fully operational, enabling any capability to be utilized on the switch with the exception of forwarding on switch panel ports. Only eth0 and console ports are activated on an un-licensed instance of Cumulus Linux. Enabling front panel ports requires a license.

You should have received a license key from Cumulus Networks or an authorized reseller. To install the license, read the Cumulus Linux quick start guide.

Upgrading to Version 1.5.3

In order to upgrade to version 1.5.3, you can either do a full upgrade of the software or, if you are upgrading from 1.5.2 to 1.5.3 only, you can use apt-get.

Warning: Save Your Configurations Before Upgrading

This installation is destructive. Any configuration files on the switch will not be saved, so please copy them to a different server before upgrading. After you save your configurations, follow these steps to upgrade to Cumulus Linux 1.5.3:

To do a full upgrade, follow these steps:

Download Cumulus Linux 1.5.3 - Final Latest Version from the Downloads page of the Cumulus Networks website onto your Web server.
Install Cumulus Linux 1.5.3, following the instructions in the quick start guide.

To use apt-get to upgrade from 1.5.2 to 1.5.3, follow these steps:

Run apt-get update.
Run apt-get upgrade.
Run apt-get install linux-image.
Reboot the switch.

To use apt-get to upgrade from 1.5.1 to 1.5.3, you need to take specific steps with smux and switchd. Follow these steps:

Before you run apt-get, you need to remove all references to smux. See Enabling Quagga below.
Run apt-get update.
Run apt-get upgrade.
Run apt-get remove switchd.
Run apt-get install bcm-utils cl-basefiles cl-platform-config uboot-tools linux-image switchd.
Reboot the switch.

Enabling Quagga

There is no SNMP support for Quagga in this release (see RN 88 below). Due to this circumstance, you must remove all references to smux in each of the following configuration files. You must also remove these references before upgrading Cumulus Linux usingapt-get. If the smux entries are present in the configuration files, the daemons in the 1.5.3 packaged version of Quagga will not start.

cd /etc/quagga
grep smux *
Delete all lines in the config files containing the smux keyword.

The references to smux that must be removed are:

In bgpd.conf, remove this line:
smux peer 1.3.6.1.4.1.3317.1.2.2 quagga_bgpd
In ospf6d.conf, remove this line:
smux peer 1.3.6.1.4.1.3317.1.2.6 quagga_ospf6d
In ospfd.conf, remove this line:
smux peer 1.3.6.1.4.1.3317.1.2.5 quagga_ospfd
In zebra.conf, remove this line:
smux peer 1.3.6.1.4.1.3317.1.2.1 quagga_zebra

What's New in 1.5.3

VRR is packaged as cl-vrr and available from the testing repo. VRR will be available in the main repo in Cumulus Linux 2.0.
Debian 7.3 update: The latest packages and security advisory patches included.

Documentation

You can read the technical documentation here.

Features Supported

The following set of features are supported in the 1.5.3 release; they're the same features that were in the 1.5.2 release.

Networking L2/L3 Features

Features	Notes
LLDP/CDP (both rx/tx)	Patched lldpd
Rapid STP	Supported via mstpd
Link Aggregation	Support provided via the Linux bonding driver
Bridging	Supported via brctl command in Linux
STP	Supported via brctl
v4/v6 DHCP relay	Supported via isc-dhcp-relay
v6 Neighbor Discovery
VLAN 802.1q trunk
ECMP
OSPFv2	Part of Quagga
OSPFv3	Part of Quagga
v4/v6 Static Routes	Part of Quagga
BGP v4/v6	Part of Quagga
Prescriptive Topology Manager	Cumulus Linux-specific user space application that allows the fabric topology to be verified prior to configuring L3 routing protocols

Security

Feature	Notes
ACL (data plane and control plane)	Support provided via the Netfilter framework, iptables, ebtables, and ip6tables, with a wrapper command called cl-acltool that you use for setting iptables and ebtables commands in the kernel and in hardware
BPDU Guard

Management Authentication/Authorization

Features	Notes
SSH
NTP
PTP

Datapath-Specific Features

Feature	Notes
Jumbo MTU
Egress scheduling	Deficit Weighted Round Robin (DWRR) + strict priority, which provides strict priority for some priorities and DWRR for the rest
Traffic classes/queuing	Classification provided via ACLs, and for 802.1p currently; DSCP-based classification provided in later release

Management Interface/Troubleshooting/Monitoring

Features	Notes
Bash-based wrapper for Quagga's vtysh	Available in /usr/bin as cl-ospf and cl-ospfv6
Scripting: Bash, Perl, Python, Ruby	Debian packages in /img.
Ping & traceroute
syslog, rsyslog
logrotate
auditd
SCP
SNMP v2 (via Net-SNMP)	Untested
Monit	Monit tested & preconfigured in the image

Issues Fixed in Cumulus Linux 1.5.3

The following is a list of issues fixed in Cumulus Linux 1.5.3 from earlier 1.5.x versions of Cumulus Linux.

Future fixed issues are noted in the Fixed Versions column with the branch name indicating when the fix will be available. If the fixed version is "mainline", this means the fix is in Cumulus Linux's internal mainline branch, but not yet allocated to a customer branch/release.

Release Note ID	Summary	Description	Affected Versions	Feature Category
RN-105	Identical SSH host key in Cumulus Linux	Identical default SSH host keys have been seen on Cumulus Linux instances. With the 1.5.3 release, a unique key is generated on the first boot of the switch.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2
RN-69	Statistics for ACL rules are reset to zero whenever the ACL rules are installed	The installation of ACL rules in hardware (using cl-acltool) is done atomically using a ping pong buffer scheme. Thus, old counters are lost any time a rule changes (is added or deleted). Thus, ACL stats for all the rules get zeroed out, irrespective of whether the rules file installation succeeds or fails. root@switch:~# iptables -L -v Chain INPUT (policy ACCEPT 291 packets, 20717 bytes) pkts bytes target prot opt in out source destination 300 134K DROP udp -- swp+ any 11.0.0.1 11.0.0.10 udp spt:800 / dpt:1000 0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:3 1 102 POLICE icmp -- swp+ any anywhere anywhere POLICE class:2 / rate:1 burst:4 0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootps / SETCLASS class:2 0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootpc / SETCLASS class:2 2 892 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match / dst-type IPROUTER POLICE class:2 rate:1 burst:4 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 242 packets, 37343 bytes) pkts bytes target prot opt in out source destination root@switch:~# cl-acltool -i -p 00control_plane.rules Using user provided rule file 00control_plane.rules Reading rule file 00control_plane.rules ... Processing rules in file 00control_plane.rules ... Installing acl policy... done. root@switch:~# iptables -L -v Chain INPUT (policy ACCEPT 8 packets, 560 bytes) pkts bytes target prot opt in out source destination 0 0 DROP udp -- swp+ any 11.0.0.1 11.0.0.10 udp spt:800 dpt:1000 0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:3 0 0 POLICE icmp -- swp+ any anywhere anywhere POLICE class:2 / rate:1 burst:4 0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootps / SETCLASS class:2 0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootpc / SETCLASS class:2 0 0 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match / dst-type IPROUTER POLICE class:2 rate:1 burst:4 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 5 packets, 612 bytes) pkts bytes target prot opt in out source destination root@switch:~#	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Security
RN-41	Route summarization acts differently in OSPFv3 vs. OSPFv2	In OSPFv2, regardless of the order of the routes, the longer prefix is always chosen: area 0.0.0.0 range 11.0.0.0/16 area 0.0.0.0 range 11.0.0.0/8 In this case, 11.0.0.0/16 is always used regardless of the input order. In OSPFv3, this is not always the case. Different results occur depending on the order routes are entered. In the following case we get the longest match: area 0.0.0.0 range 2000:1000::/32 area 0.0.0.0 range 2000::/16 However, if we do it in the reverse order of: area 0.0.0.0 range 2000::/16 area 0.0.0.0 range 2000:1000::/32 You get both summarizations. This has been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Layer 3 (Routing)
RN-71	SPAN: Cannot match an output L3 sub-interface	Mirroring packets from one switch port to another works: -A FORWARD --out-interface swp1 -j SPAN --dport swp5 As does mirroring inbound packets on a VLAN to a switch port: -A FORWARD --in-interface swp1.100 -j SPAN --dport swp5 However, mirroring outbound packets from a VLAN to a switch port does not work: -A FORWARD --out-interface swp1.100 -j SPAN --dport swp5 In summary: * Matching output L3 main interface works fine. * Matching input L3 sub-interface works fine. * Matching output L3 sub-interface does not work. This has been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Layer 2 (Segmentation)
RN-106	STP interoperability	Cumulus Linux supports RSTP/PVRST/PVST modes of STP natively. Cumulus Linux can interoperate with Common Spanning Tree/Multiple Spanning Tree Protocol implementations in other vendors.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Layer 2 (Segmentation)
RN-12	The switch's forwarding of VLAN-tagged packets is different from Linux	In Cumulus Linux, tagged packets sent to an untagged port get dropped. This is similar to general switch functionality from most vendors. However, in Linux, if a tagged packet is sent to an untagged port, it gets forwarded. This has been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Layer 2 (Segmentation)
RN-53	In OSPFv3, a new router LSA may not be generated if the neighbor advertises a router LSA with a different interface ID	If the neighboring router has the interface ID change in the Hello message, Quagga will not generate an update LSA. This has been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Layer 3 (Routing)
RN-110	SNMP ifIndex is Cumulus OID instead of Linux OID	SNMP ifIndex is Cumulus OID instead of Linux OID. This has been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Monitoring
RN-81	Hardware enforcement for bridging packets that match both ebtables and iptables/ip6tables rules have different behavior from Linux kernel enforcement	iptables Issue: The behavior for bridging packets that match the rules for both ebtables and iptables is different from netfilter enforcement for packets matching these tables in the Linux kernel. ebtables are always enforced before iptables for bridging packets, whereas iptables are enforced for hardware. ip6tables Issue: The behavior for bridging packets that match the rules for both ebtables as well as ip6tables is different from netfilter enforcement for packets matching these tables in the Linux kernel. ebtables are always enforced before ip6tables for bridging packets in the Linux kernel, whereas the actions in both these tables (ebtables/ip6tables) for the packet will try to be enforced. These issues have been fixed with this release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2	Security

Known Issues in Cumulus Linux 1.5.3

Issues are categorized for easy review. Some issues are fixed but will be available in a later release.

Release Note ID	Summary	Description	Affected Versions	Feature Category
RN-1	Restarting switchd flaps all switch ports	switchd is a user-level process created by Cumulus Networks to provide an abstraction of the physical ports and the functionality provided by the switching ASIC SDK. switchd maps physical ports on the switching ASIC to logical ports (tap ports) in the kernel and ensures that CPU-bound packets are properly exposed on the proper logical objects to user level processes. These exposed tap ports in the kernel are considered "running" if their file descriptors are open. If switchd exists, it closes the tap FDS, hence resulting in all links going down.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Hardware Abstraction Layer
RN-4	ifup/ifdown must be used for interfaces with IPv6 addresses defined in /etc/network/interfaces, otherwise the interfaces will be missing their IPv6 addresses	Two scenarios are shown below; one with ifup/ifdown, the other with ifconfig down. With ifup/ifdown: swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4231 errors:0 dropped:0 overruns:0 frame:0 TX packets:4342 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:412115 (402.4 KiB) TX bytes:425688 (415.7 KiB) root@switch$ ifdown swp1 root@switch$ ifconfig swp1 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:4248 errors:0 dropped:0 overruns:0 frame:0 TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB) root@switch$ ifconfig swp1 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:4248 errors:0 dropped:0 overruns:0 frame:0 TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB) root@switch$ ifup swp1 ADDRCONF(NETDEV_UP): swp1: link is not ready root@switch$ ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: link / becomes ready swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4250 errors:0 dropped:0 overruns:0 frame:0 TX packets:4362 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:414178 (404.4 KiB) TX bytes:427610 (417.5 KiB) root@switch$ With ifconfig down: ifconfig swp1 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:98 errors:0 dropped:0 overruns:0 frame:0 TX packets:111 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:13310 (12.9 KiB) TX bytes:12786 (12.4 KiB) root@switch$ ifconfig swp1 down root@switch$ ifconfig swp1 swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:126 errors:0 dropped:0 overruns:0 frame:0 TX packets:138 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:16998 (16.5 KiB) TX bytes:15998 (15.6 KiB) root@switch$ ifconfig swp1 up ADDRCONF(NETDEV_UP): swp1: link is not ready root@switch$ ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: link / becomes ready swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81 inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:130 errors:0 dropped:0 overruns:0 frame:0 TX packets:149 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:17474 (17.0 KiB) TX bytes:17154 (16.7 KiB)	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Management Interface
RN-10	cl-phy-update doesn't support aggregated ports	Ports can be aggregated into a larger interface in Cumulus Linux. Unfortunately support for aggregated ports is not yet supported when running cl-phy-update. If there are any ganged ports during a software upgrade, it is recommended to ungang these ports.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Tools
RN-32	Adding bridges increases bootup time	If the bridge_maxwait parameter is not set, the system will take approximately twice as long to come up. You should set bridge_maxwait to 1. Example config: auto br1004 iface br1004 inet static address 14.0.0.37 netmask 255.255.0.0 bridge_ports regex (swp[1\|6\|7\|8].1004) bridge_stp on bridge_bridgeprio 32768 bridge_maxwait 1 bridge_ageing 200 bridge_fd 30 down ip addr flush dev br1004	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 2 (Control Protocols)
RN-48	Agema 48x10GE switch eth0 driver reports eth0 as running even when PHY link is down	The Agema 48x10GE eth0 driver reports eth0 as running even when the PHY link is down. This can be really misleading in trying to diagnose a link-down situation on eth0. ethtool eth0 shows the correct PHY link status, but ifconfig shows eth0 as running, regardless of the PHY link status. A fix will be released after Cumulus Linux 1.5.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Management Interface
RN-52	Parameters like the router ID and DR priority cannot be changed while OSPFv2/v3 is running	Router ID and DR priority can only be changed by shutting down OSPFv2/v3, changing the ID, and restarting the OSPF process. A change to the DR priority may not properly be reflected in the LSAs that are still aging out.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-56	ipv4/ipv6 forwarding disabled mode not recognized	If either of the following is configured: net.ipv4.ip_forward == 0 or: net.ipv6.conf.all.forwarding == 0 The hardware still forwards packets if there is a neighbor table entry pointing to the destination.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Forwarding)
RN-58	IPv6 route is installed and active in the routing table when the associated interface is down	If an IPv6 address is assigned to a "down" interface, the associated route is still installed into the route table. Also, the type of IPv6 address doesn't matter. Link local, site local, and global all exhibit the same problem. If the interface is bounced up and down, then the routes are no longer in the route table.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Forwarding)
RN-61	BGP4 notifications missing for several conditions	In certain conditions, Quagga bgpd silently closes the peering without sending a notification. For example, if BGP receives a message with an invalid message type or invalid message length. Ideally on any one of these cases, bgpd should send out a notification message to the peer. General functionality of BGP4 is not affected.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-62	Attributes of a BGP aggregate route may not be RFC-compliant	When BGP is configured with an aggregate route and there are more specific routes of that aggregate, the BGP speaker needs to analyze the attributes of those specific routes while forming the attributes of the aggregate route. The corresponding rules are defined in RFC 4271, Sect. 9.2.2.2. In certain cases, the user may observe non-compliant attribute formation for the aggregate route: for example, incorrect MED and ORIGIN attributes.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-63	BGP4 recursive route not supported	Quagga's bpgd does not support recursive routing when it's resolving BGP routes. It does work with respect to IGP routes. As an example: A BGP route of the form: R1 -> N1 where the best match for N1 is also a BGP route: N1/24 -> N2 (and N2 gets resolved through IGP or directly connected) - will not get resolved. Quagga's Zebra specifically does not allow for such types of routes. The Zebra route path resolution algorithm needs to be fixed. Although, Cumulus Linux's version of Quagga -- 0.99.21 -- has errors in recursive routing, the community is actively trying to fix it and patches are being tested in 0.99.22.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-64	Configuring route-reflector-client requires specific order	In configuring a route to be a route reflector client, the Quagga configuration must be specified in a specific order; otherwise, the router will not be a route reflector client. The "neighbor <IPv4/IPV6> route-reflector-client" command must be done after the "neighbor <IPV4/IPV6> Activate" command; otherwise, the route-reflector-client command is ignored. Sample configuration: router bgp 65000 bgp router-id 0.0.0.4 bgp log-neighbor-changes no bgp default ipv4-unicast bgp cluster-id 0.0.0.4 bgp bestpath as-path multipath-relax redistribute connected neighbor 14.0.0.1 remote-as 65000 neighbor 14.0.0.1 route-reflector-client neighbor 14.0.0.1 activate neighbor 14.0.0.1 next-hop-self neighbor 14.0.0.9 remote-as 65000 neighbor 14.0.0.9 activate neighbor 14.0.0.9 next-hop-self neighbor 2001:ded:beef::1 remote-as 65000 neighbor 2001:ded:beef:2::1 remote-as 65000 maximum-paths 4 maximum-paths ibgp 4 ! address-family ipv6 redistribute connected neighbor 2001:ded:beef::1 activate neighbor 2001:ded:beef::1 next-hop-self neighbor 2001:ded:beef:2::1 route-reflector-client neighbor 2001:ded:beef:2::1 activate neighbor 2001:ded:beef:2::1 next-hop-self maximum-paths 4 maximum-paths ibgp 4 exit-address-family At runtime: sp1# show ip bgp neighbor 14.0.0.1 BGP neighbor is 14.0.0.1, remote AS 65000, local AS 65000, / internal link BGP version 4, remote router ID 0.0.0.6 BGP state = Established, up for 00:23:49 Last read 23:31:36, hold time is 180, keepalive interval / is 60 seconds Neighbor capabilities: 4 Byte AS: advertised and received Route refresh: advertised and received(old & new) Address family IPv4 Unicast: advertised and received Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 2 0 Notifications: 0 0 Updates: 1 1 Keepalives: 25 24 Route Refresh: 0 0 Capability: 0 0 Total: 28 25 Minimum time between advertisement runs is 5 seconds For address family: IPv4 Unicast >>>>>>>>>>>>>>>>>>>>>> ROUTE REFLECTOR CLIENT NOT DISPLAYED NEXT_HOP is always this router Community attribute sent to this neighbor(both) 6 accepted prefixes Connections established 1; dropped 0 Last reset never Local host: 14.0.0.2, Local port: 179 Foreign host: 14.0.0.1, Foreign port: 40290 Nexthop: 14.0.0.2 Nexthop global: 2001:ded:beef::2 Nexthop local: fe80::202:ff:fe00:4 BGP connection: non shared network Read thread: on Write thread: off sp1# Workaround: Define in following order address-family ipv4 unicast neighbor 14.0.0.9 activate neighbor 14.0.0.9 next-hop-self neighbor 14.0.0.9 route-reflector-client >>> Must be / after Activate exit-address-family neighbor 2001:ded:beef:2::1 remote-as 65000 address-family ipv6 unicast redistribute connected maximum-paths 4 maximum-paths ibgp 4 neighbor 2001:ded:beef:2::1 activate neighbor 2001:ded:beef:2::1 next-hop-self neighbor 2001:ded:beef:2::1 route-reflector-client >>> / Must be after activate exit-address-family Runtime status after change: sp1# show ip bgp neighbors 14.0.0.9 BGP neighbor is 14.0.0.9, remote AS 65000, local AS 65000, / internal link BGP version 4, remote router ID 0.0.0.7 BGP state = Established, up for 00:13:59 Last read 22:35:13, hold time is 180, keepalive interval / is 60 seconds Neighbor capabilities: 4 Byte AS: advertised and received Route refresh: advertised and received(old & new) Address family IPv4 Unicast: advertised and received Message statistics: Inq depth is 0 Outq depth is 0 Sent Rcvd Opens: 1 1 Notifications: 0 0 Updates: 2 1 Keepalives: 15 14 Route Refresh: 0 0 Capability: 0 0 Total: 18 16 Minimum time between advertisement runs is 5 seconds For address family: IPv4 Unicast Route-Reflector Client >>>>>>>>>> PLEASE NOTE ME NEXT_HOP is always this router Community attribute sent to this neighbor(both) 6 accepted prefixes Connections established 1; dropped 0 Last reset never Local host: 14.0.0.10, Local port: 38813 Foreign host: 14.0.0.9, Foreign port: 179 Nexthop: 14.0.0.10 Nexthop global: 2001:ded:beef:2::2 Nexthop local: fe80::202:ff:fe00:6 BGP connection: non shared network Read thread: on Write thread: off sp1#	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-65	Virtual links in Quagga's OSPFv2 are non-operational	Cumulus Networks testing has identified too many issues with virtual link support in Quagga's OSPFv2. The feature is unsupported.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-68	Blackhole/Unreachable/Prohibit route addition in IPv6 returns corresponding error codes	IPv6 route operations indicate the destination action via returned error codes. In the example shown below where an unreachable route is being added, the return code is: #define ENETUNREACH 101 /* Network is unreachable */ ip addr add 9000:1000:1000:1000::1/80 dev lo root@switch$ ip -6 route unreachable 9000:1000:1000:1000::/80 dev lo proto kernel / metric 256 error -101	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Forwarding)
RN-70	ACL: Bridge traffic that matches a LOG ACTION rule is not logged in syslog	For example, a bridge with switch ports swp1, swp2, swp3 as bridge members is configured. ACL rules to LOG and DROP for icmp traffic are configured. Ping requests are sent from host1 on swp1 to host3 on swp3, and the following was observed: * Counters for both LOG and DROP ACL rules are incrementing properly, but the packets are not showing up on /var/log/syslog. * Packets that are copied to the CPU from hardware for the LOG rule are dropped due to the check in kernel to disable software bridging for hardware bridged packets.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Security
RN-74	BGP: TTL security hops requires ACL per peer to drop expired TTL BGP packers	When configuring TTL security, rules must be put in to drop packets with exceeded TTL on BGL peers. This ensures that when the peers are taken down, the packets are properly discarded. Here is an example of a rule to put in iptables: root@switch:~# cat /etc/cumulus/acl/policy.d/bgp.rules INGRESS_INTF = swp+ [iptables] -A INPUT --in-interface $INGRESS_INTF -p tcp --dport bgp / --src 11.0.0.6 -m ttl --ttl-eq 253 -j POLICE --set-mode / pkt --set-rate 2000 --set-burst 1000 --set-class 7	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Routing)
RN-77	New routes/ECMPs can evict existing/installed	Cumulus Linux syncs routes between the kernel and the switching silicon. If the required resource pools in hardware fill up, new kernel routes can cause existing routes to move from being fully allocated to being partially allocated. In order to avoid this, routes in the hardware should be monitored and kept below the ASIC limits. For example, on systems with Trident+, the limits are as follows: routes: 16384 <<<< if all routes are ipv4 long mask routes 256 <<<< i.e., routes with a mask longer than the route mask limit route mask limit 64 host_routes: 8192 ecmp_nhs: 4044 ecmp_nhs_per_route: 52 That translates to about 77 routes with ECMP NHs, if every route has the maximum ECMP NHs. Monitoring this in Cumulus Linux is performed via the cl-resource-query command: root@switch:~# cl-resource-query hosts : 3 all routes : 29 IP4 routes : 17 IP6 routes : 12 nexthops : 3 ecmp_groups : 0 ecmp_nexthops : 0 mac entries : 0 / 131072 bpdu entries : 500 / 512 The resource to monitor is the ecmp_nexthops. If this count is close to 4044, new ECMPs may evict existing routes.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 3 (Forwarding)
RN-80	Old link local IPv6 and MAC address are still used after the MAC address for a bridge changes	Old MAC addresses for a bridge using the associated link local IPv6 address are used in the L2 and L3 headers in a router advertisement after the port is removed from the bridge. To reproduce this: 1. Configure a bridge, br0, with swp1-4 in it. 2. Remove swp1 from br0 with the command "brctl delif br0 swp1" 3. Run a TCPDump session on swp2. The TCPDump session reveals that the old MAC address, swp1's MAC address, and link local address are still being advertised. Here is a packet from the capture: 19:41:15.041412 44:38:39:00:01:81 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::4638:39ff:fe00:181 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24 hop limit 64, Flags [none], pref medium, router lifetime 15s, reachable time 0s, retrans time 0s source link-address option (1), length 8 (1): 44:38:39:00:01:82 0x0000: 4438 3900 0182 44:38:39:00:01:81 was the old MAC address for bridge br0 that was associated with swp1. fe80::4638:39ff:fe00:181 is the link local address for swp1.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 2 (Segmentation)
RN-85	Error log messages displayed with cl-resource-query when following a switchd restart	If you use cl-resource-query following a switchd restart, a transient hardware error may occur and trigger several error messages. Based on current analysis, the error is harmless, and Cumulus Networks will work to remove it in a future release.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Tools
RN-88	SNMP Support for Quagga Is NOT Provided in Cumulus Linux	Cumulus Linux 1.5.3 does not provide SNMP support for Quagga.	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Monitoring
RN-91	sfputil Outputs Garbled Data for Optical QSFP on a Penguin 48x10GE	The following example shows the garbled output from sfputil for a 10GE port: root@switch:/sys/class/eeprom_dev/eeprom54/device# cl-sfputil / -p swp52 swp52: SFP detected Connector : MT-RJ EncodingCodes : Unknown ExtIdentOfTypeOfTransceiver : Unknown Length50um(UnitsOf10m) : 66 Length62.5um(UnitsOfm) : 132 LengthCable(UnitsOfm) : 8 LengthOM3(UnitsOf10m) : 16 LengthSMFkm-UnitsOfKm : 128 NominalSignallingRate(UnitsOf100Mbd) : 32 RateIdentifier : Unknown ReceivedPowerMeasurementType : OMA TransceiverCodes : 10GEthernetComplianceCode : 10G Base-SR EthernetComplianceCodes : BASE-PX FibreChannelTechnology : Shortwave laser w/o OFC (SN) FibreChannelTransmissionMedia : Multimode, 50um (M5, M5E) SONETComplianceCodes : OC-12, single mode, long reach TypeOfTransceiver : Unknown VendorDataCode(yymmdd) : ??B?1b VendorName : @?B @?B? VendorOUI : @? VendorPN : B @?B @? VendorRev : B? VendorSN : B?M?4hРB?0`	CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3
RN-92	Quagga Does not Send Open Message when Passively Accepting an Open Connection Request	As per RFC4271, an open request must be initiated back to the initiating (active) side of the connection request. Quagga does not send an open request upon receiving an active open request. The work around is to set the timers connect to "1". Here is an example: router1 <-------------------> router2 router1 router bgp 3500100001 neighbor 13.0.0.13 remote-as 3400100001 neighbor 13.0.0.13 timers connect 1 neighbor 13.0.0.13 activate router2 router bgp 3400100001 neighbor 13.0.0.14 remote-as 3500100001 neighbor 13.0.0.14 passive >>>>>>>>>>>PLEASE NOTE neighbor 13.0.0.14 timers connect 1 neighbor 13.0.0.14 activate	CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3
RN-99	cl-img-clear-overlay Is Disabled if Kernel Is Upgraded Using apt-get	If you upgraded the kernel to version 1.5.3 using apt-get update, then cl-img-clear-overlay will be disabled. To ensure Cumulus Linux and all its contained packages are in sync, and to be able to use cl-img-clear-overlay, perform a full install of Cumulus Linux using cl-img-install.	CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3
RN-102	Error when Configuring VXLAN Network ID 0	Problem: vni0 is allowed but an error occurs. Steps to reproduce: In script, set the VNI ID as 0 for the VXLAN netdev device. 41: vxLan_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-1-0 state UNKNOWN mode DEFAULT link/ether da:2c:2a:7e:05:ea brd ff:ff:ff:ff:ff:ff vxlan id 0 local 12.0.0.17 port 32768 61000 ageing 300 ^^ Not ME 42: br-1-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether 44:38:39:00:44:82 brd ff:ff:ff:ff:ff:ff bridge Form the bridge. Try to add a MAC address. The following error occurs: In [25]: pp.pprint(list_me) ['1381369733.607298 2013-10-10 01:48:53 hal_bcm.c:1130 CRIT Ticket: CM-1573: DROP_PKT_CNT_INGr counter not supported on TRIDENT2', '1381369812.155459 2013-10-10 01:50:12 hal_bcm.c:1130 CRIT Ticket: CM-1573: DROP_PKT_CNT_INGr counter not supported on TRIDENT2', '1381370042.649162 2013-10-10 01:54:02 hal_bcm.c:1130 CRIT Ticket: CM-1573: DROP_PKT_CNT_INGr counter not supported on TRIDENT2', '1381370281.875083 2013-10-10 01:58:01 hal_bcm.c:1130 CRIT Ticket: CM-1573: DROP_PKT_CNT_INGr counter not supported on TRIDENT2', '1381370286.427230 2013-10-10 01:58:06 hal_bcm_vxlan.c:381 CRIT VPN create failed: -8', <<?? '1381370286.978184 2013-10-10 01:58:06 hal_bcm_vxlan.c:381 CRIT VPN create failed: -8', '1381370287.114601 2013-10-10 01:58:07 hal_bcm_vxlan.c:381 CRIT VPN create failed: -8', '1381370287.383625 2013-10-10 01:58:07	mainline
RN-103	In a VRR environment, the server that is bonded to the VRR switches could lose packets destined to the VRR's IP addresses for up to 15 seconds.	For the following configuration: . r1 . / \ . vrr1------vrr2 . \ / . host1 The hosts have bond interfaces where one sub-interface goes to switch vrr1, and the other goes to the other switch, vrr2. If the link between the host and one of the VRR switches goes down, it can take up to 15 seconds of the VRR switches to send out an ARP to clear the ARP cache on the host for the IP address on the bridge interface. This is because the host might not clear the ARP cache since the bond doesn't go down. Only a sub-interface in the bond goes down. Steps to reproduce: One of the the hosts connected to the VRR switches, ping the real IP addresses of the bridge. On the same host, bring the active interface down with `ip link set <interface> down` and let the backup take over. Ping the real IP addresses of the VRR switch that is connected to the active interface.	CumulusLinux-1.5.3	Layer 2 (Control Protocols)
RN-104	Bond slave dynamic updates are not working in bridge bond case.	You should update `/etc/network/interfaces` and restart network services in order to add/remove slave interface from bond. Using other methods such as editing `/sys/class/net/bondX/bonding` will result in the added or removed slave interface to not forward traffic correctly. Initial condition: root@dni-7448-05:~# brctl show bridge name bridge id STP enabled interfaces br0 8000.443839000385 yes bond2 bond3 br2000 8000.443839000385 yes bond2.2000 bond3.2000 br2001 8000.443839000385 yes bond2.2001 bond3.2001 br2002 8000.443839000385 yes bond2.2002 bond3.2002 br2003 8000.443839000385 yes bond2.2003 bond3.2003 root@dni-7448-05:~# cat /tmp/bridge_info br0:2000 br2000:2001 br2001:2002 br2002:2003 br2003:2004 root@dni-7448-05:~# root@dni-7448-05:~# cat /proc/net/bonding/bond3 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Min links: 1 Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 33 Partner Key: 33 Partner Mac Address: 00:02:00:00:00:1e Slave Interface: swp7 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 44:38:39:00:03:87 Aggregator ID: 1 Slave queue ID: 0 Slave Interface: swp8 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 44:38:39:00:03:88 Aggregator ID: 1 Slave queue ID: 0 root@dni-7448-05:~# ===== root@dni-7448-05:~# mstpctl settreeprio br0 0 61440 root@dni-7448-05:~# mstpctl showbridge br0 br0 CIST info enabled yes bridge id F.000.44:38:39:00:03:85 designated root 2.000.00:02:00:00:00:1A regional root F.000.44:38:39:00:03:85 root port bond2 (#1) path cost 1000 internal path cost 0 max age 20 bridge max age 20 forward delay 15 bridge forward delay 15 tx hold count 6 max hops 20 hello time 2 ageing time 300 force protocol version rstp time since topology change 8s topology change count 2 topology change no topology change port bond2 last topology change port bond3 root@dni-7448-05:~# mstpctl showport br0 bond2 8.001 forw 2.000.00:02:00:00:00:1A / 2.000.00:02:00:00:00:1A 8.001 Root bond3 8.002 disc 2.000.00:02:00:00:00:1A / 8.000.00:02:00:00:00:1E 8.001 Altn root@dni-7448-05:~# root@dni-7448-05:~# /usr/lib/cumulus/bcmcmd stg show STG 1: contains 56 VLANs (1,2901-2903,3901-3952) Disable: xe0-xe1,xe4-xe51 Forward: xe2-xe3 STG 2: contains 1 VLAN (2900) Disable: xe2-xe51 Forward: xe0-xe1 STG 3: contains 1 VLAN (2000) Disable: xe0-xe3,xe8-xe51 Block: xe6-xe7 >>> Life is perfect Forward: xe4-xe5 STG 4: contains 1 VLAN (2904) Disable: xe STG 5: contains 1 VLAN (2905) Disable: xe STG 6: contains 1 VLAN (2001) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>> life is perfect STG 7: contains 1 VLAN (2906) Disable: xe STG 8: contains 1 VLAN (2907) Disable: xe STG 9: contains 1 VLAN (2002) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>> life is perfect STG 10: contains 1 VLAN (2908) Disable: xe STG 11: contains 1 VLAN (2909) Disable: xe STG 12: contains 1 VLAN (2003) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>> life is perfect STG 13: contains 1 VLAN (2910) Disable: xe STG 14: contains 1 VLAN (2911) Disable: xe STG 15: contains 1 VLAN (2004) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>> life is perfect root@dni-7448-05:~# {noformat} {noformat} Trigger: ---------------- Now Lets remove the bond port root@dni-7448-05:~# echo '-swp8' > / /sys/class/net/bond3/bonding/slaves root@dni-7448-05:~# cat /sys/class/net/bond3/bonding/slaves swp7 root@dni-7448-05:~# +++++++++++++++++++++++++++ root@dni-7448-05:~# /usr/lib/cumulus/bcmcmd stg show STG 1: contains 56 VLANs (1,2901-2903,3901-3952) Disable: xe0-xe1,xe4-xe6,xe8-xe51 Block: xe7 Forward: xe2-xe3 STG 2: contains 1 VLAN (2900) Disable: xe2-xe51 Forward: xe0-xe1 STG 3: contains 1 VLAN (2000) Disable: xe0-xe3,xe7-xe51 Block: xe6 >>>> Correct amd updated Forward: xe4-xe5 STG 4: contains 1 VLAN (2904) Disable: xe STG 5: contains 1 VLAN (2905) Disable: xe STG 6: contains 1 VLAN (2001) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>>> Not Updated ??? STG 7: contains 1 VLAN (2906) Disable: xe STG 8: contains 1 VLAN (2907) Disable: xe STG 9: contains 1 VLAN (2002) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>>> Not Updated ??? STG 10: contains 1 VLAN (2908) Disable: xe STG 11: contains 1 VLAN (2909) Disable: xe STG 12: contains 1 VLAN (2003) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>>> Not Updated ??? STG 13: contains 1 VLAN (2910) Disable: xe STG 14: contains 1 VLAN (2911) Disable: xe STG 15: contains 1 VLAN (2004) Disable: xe0-xe3,xe8-xe51 Forward: xe4-xe7 >>>> Not Updated ??? root@dni-7448-05:~#	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 2 (Segmentation)
RN-108	When interoperating with Cisco Rapid-PVST, use trunk VLAN ID 1 only	This problem occurs during IOP environment where a peer like Cisco or Arista is connected and Rapid PVST mode is enabled. In such an environment, if you connect a VLAN trunk link to a Cumulus Linux switch that has a native VLAN ID other than 1, IOP fails. Cisco implemented SSTP packets to perform the handshake between IEEE-802.1w or standard RSTP. This is the packet format: 22:10:35.929814 44:38:39:00:03:85 (oui Unknown) > 01:00:0c:cc:cc:cd (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 2000, p 0, LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid PVST (0x010b): STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 1000.44:38:39:00:03:85.8001, length 42 message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 15.00s root-id 1000.44:38:39:00:03:85, root-pathcost 0, port-role Designated 0x0000: 0100 0ccc cccd 4438 3900 0385 8100 07d0 0x0010: 0032 aaaa 0300 000c 010b 0000 0202 7c10 0x0020: 0044 3839 0003 8500 0000 0010 0044 3839 0x0030: 0003 8580 0100 0014 0002 000f 0000 T:0000 0x0040: L:0002 V:07d0 In the TLV shown above, V indicate the VLAN ID. If native VLAN ID is 1, then Cisco/Arista would generate the IEEE STP PDU instate of SSTP PDU. From the Cumulus Linux perspective, in a given bridge if an interface is not a sub-interface, then it generates the IEEE STP PDU. With a sub-interface, say swp2.100 or bond1.1000, it generates the SSTP PDU. Cisco would send SSTP PDU when the native VLAN ID is not 1. Cumulus Linux does not consider PVID marking today. Thus, the system would get PVID and type inconsistencies check failure on their product. (See http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00801d11a0.shtml.)	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Layer 2 (Segmentation)
RN-109	switchd restarts in rare cases when missing heartbeat	Under a rare and unusual condition, when a switch experiences a parity error, `switchd` will restart. To work around this issue, add the following option to `/etc/default/switchd` and to `/mnt/persist/etc/default/switchd`: `DAEMON_OPT_ARGS="$DAEMON_OPT_ARGS -g"`	CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1, CumulusLinux-1.5.2, CumulusLinux-1.5.3	Hardware Abstraction Layer
RN-313	High memory utilization by `snmpd following` MIB walks	When performing an MIB walk on Cumulus Linux running the standard NET-SNMP 5.4.3 package, memory is consumed and not returned. Over time, this can lead to sub-optimal performance and ultimately can cause the system to hang and require a reboot. Monitor `snmpd` memory utilization using `ps aux` and system memory utilization with `free` or `cat /proc/meminfo`. If required, restart snmpd using `service snmpd restart`.
RN-372 (CM-9360)	Security Update for CVE-2015-7547: glibc getaddrinfo Stack-based Buffer Overflow Vulnerability	For details on this issue and how to upgrade, read this article.

Release Notes for Cumulus Linux 1.5.3-Final