Release Notes for Cumulus Linux 1.5.1-Final

Follow

Overview

These release notes support Cumulus Linux 1.5.1 and describe currently available features, and known issues. 

Licensing

Cumulus Linux is licensed on a per-instance basis. Each network system is fully operational, enabling any capability to be utilized on the switch with the exception of forwarding on switch panel ports. Only eth0 and console ports are activated on an un-licensed instance of Cumulus Linux. Enabling front panel ports requires a license.

You should have received a license key from Cumulus Networks or an authorized reseller. To install the license, read the Cumulus Linux quick start guide

Upgrading to Version 1.5.1

In order to upgrade to version 1.5.1, you need to do a full upgrade of the software; apt-get will not work.

Warning: Save Your Configurations Before Upgrading

This installation is destructive. Any configuration files on the switch will not be saved, so please copy them to a different server before upgrading. After you save your configurations, follow these steps to upgrade to Cumulus Linux 1.5.1: 

  1. Download Cumulus Linux 1.5.1 - Final Latest Version from the Downloads page of the Cumulus Networks website onto your Web server.
  2. Install Cumulus Linux 1.5.1, following the instructions in the quick start guide.
Enabling L3 Routing

There is no SNMP support for Quagga in this release (see RN 88 below). Due to this change, you must remove the line listed in each of the following Quagga configuration files in /etc/quagga:

  • In bgpd.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.2 quagga_bgpd
  • In ospf6d.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.6 quagga_ospf6d
  • In ospfd.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.5 quagga_ospfd
  • In zebra.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.1 quagga_zebra

If these lines are not removed, the Quagga daemons listed here will not start.

What's New in 1.5.1

  • Cumulus Linux 1.5.1 now supports the following two platforms:
Manufacturer Part Number Description
Penguin Arctica 4804i 48 x 1G-T & 4 x 10G-SFP+
Penguin Arctica 4804x 48 x 10G-SFP+ & 4 x 40G-QSFP+
  • Zero-touch provisioning: Provides the ability to automate configuration of the network OS through discovery of a user-provided script (via a DHCP option). You can also use this to install and configure third party automation tools, like Puppet.

Documentation

You can read the technical documentation here.

Features Supported

The following set of features are supported in the 1.5.1 release; they're the same features that were in the 1.5.0 release.

Networking L2/L3 Features

Feature Notes
LLDP/CDP (both rx/tx) Patched lldpd
Rapid STP Supported via mstpd
Link Aggregation Support provided via the Linux bonding driver
Bridging Supported via brctl command in Linux
STP Supported via brctl
v4/v6 DHCP relay Supported via isc-dhcp-relay
v6 Neighbor Discovery  
VLAN 802.1q trunk  
ECMP  
OSPFv2 Part of Quagga
OSPFv3 Part of Quagga
v4/v6 Static Routes Part of Quagga
BGP v4/v6 Part of Quagga
Prescriptive Topology Manager Cumulus Linux-specific user space application that allows the fabric topology to be verified prior to configuring L3 routing protocols

Security

Feature Notes
ACL (data plane and control plane) Support provided via the Netfilter framework, iptables, ebtables, and ip6tables, with a wrapper command called cl-acltool that you use for setting iptables and ebtables commands in the kernel and in hardware
BPDU Guard  

Management Authentication/Authorization

Feature Notes
SSH  
NTP  
PTP  

 

Datapath-Specific Features

Feature Notes
Jumbo MTU  
Egress scheduling Deficit Weighted Round Robin (DWRR) + strict priority, which provides strict priority for some priorities and DWRR for the rest
Traffic classes/queuing Classification provided via ACLs, and for 802.1p currently; DSCP-based classification provided in later release

Management Interface/Troubleshooting/Monitoring

Feature Notes
Bash-based wrapper for Quagga's vtysh Available in /usr/bin as cl-ospf and cl-ospfv6
Scripting: Bash, Perl, Python, Ruby Debian packages in /img.
Ping & traceroute  
syslog, rsyslog  
logrotate  
auditd  
SCP  
SNMP v2 (via Net-SNMP) Untested
Monit Monit tested & preconfigured in the image

 

Issues Fixed in Cumulus Linux 1.5.1

The following is a list of issues fixed in Cumulus Linux 1.5.1 from Cumulus Linux 1.5.0

Key Summary Description affectedTag fixedTag Feature Category
RN-78 Ensure configurations are backed up in /mtn/persist prior to upgrades Cumulus Linux provides a way to ensure your configurations are persistent across upgrades. All files needs to be copied into /mnt/persist in order to ensure they are persistent across upgrades.

Check out article at https://support.cumulusnetworks.com/entries/24087796-Copying-configurations-across-switches-e-g-bad-switch-to-new-switch- for more information.
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Configuration Management
RN-72 LLDP refresh time should not be set below a specific amount; otherwise, entries will be deleted before the timer expires lldp settings for hold and interval should be:

root@switch:~# cat /etc/lldpd.conf
configure lldp tx-interval 1
configure lldp tx-hold 4
root@switch:~#

If these are set to below 1 and 4 respectively, entries will be deleted before the timer expires.

root@switch:~# lldpcli show neighbors hidden details -f keyvalue | grep via=LLDP
lldp.swp4.via=LLDP <<< Entries are removed


Following are the expected entries.

root@switch:~# lldpcli show neighbors hidden details -f keyvalue | grep via=LLDP
lldp.swp1.via=LLDP
lldp.swp2.via=LLDP
lldp.swp3.via=LLDP
lldp.swp4.via=LLDP
lldp.swp5.via=LLDP
lldp.swp6.via=LLDP
lldp.swp7.via=LLDP

If you want the neighbor to retain your port, you should always have txTTL > txInterval.
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Layer 2 (Control Protocols)
RN-31 IP packets with illegal source addresses are forwarded IP packets with the illegal source addresses of 127.0.0.1, 255.255.255.255, and the subnet broadcast are now forwarded as if the source addresses were legal. RFC1812 states that packets with these source addresses should not be forwarded. CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-59 BGP should set the correct next-hop based on address-family of NLRIs being advertised Currently BGP will utilize the IPv4 next hop address for both IPv4 and IPv6 NLRIs.

In essence, for each set of NLRIs, the next hop address should be picked up based on the peering specifics; that is, if interface peering, pick the v4 or v6 next-hop of the interface. If loopback peering (or update-source), pick the v4 or v6 next-hop of the loopback.

This should also work if an outbound route map is configured with the statement "set ip next-hop peer-address".
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Layer 3 (Routing)
RN-60 Interface bringup errors occur during network restart If the /etc/network/interfaces file uses the vlan_raw_device command as part of a VLAN interface definition, errors during network bringup (service network restart) or reboot will occur. These errors should be ignored. (See below.)

For instance, if the /etc/network/interfaces file has:

auto swp1.1001
iface swp1.1001 inet static
address 11.0.1.81
netmask 255.255.255.240
vlan_raw_device swp1
up vconfig set_egress_map swp1.1001 0 1
down ip addr flush dev swp1.1001


Then, while the network restarts (or reboot), the following error occurs:
------------------------------
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
ERROR: trying to add VLAN #1001 to IF -:swp1:- error: File exists <<IGNORE THIS ERROR
Set egress mapping on device -:swp1.1001:- Should be visible in /proc/net/vlan/swp1.1001

Essentially, iface swp1.1001 inet static brings the interface up, and vlan_raw_device also tries to bring it up (via vconfig), even though its already up. Hence the error.

This error also occurs if the interface is a bonding interface.
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Management Interface
RN-54 "!" in the "-p" option is not supported for iptables Cumulus Linux doesn't support the following list of options: 
"!" in the "-p" option
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Security
RN-82 Byte count for ebtables is off by 4 for untagged packets When a L2 packet matches a rule in ebtables, the byte counter will be off by 4 bytes when an untagged packet was used to cause the match. This is due to the fact that Cumulus Linux appends a VLAN tag onto a packet once the packet enters the device. If the packet is VLAN tagged coming into the device, then the packet count will be correct. CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Security
RN-76 cl-route-check will fail when checking routes against bridges with IP addresses and bonding interfaces If you assign an IP address to a bridge interface, cl-route-check will say that the route for this address wasn't found in the hardware tables. Similarly, any routes against a bonding interface will fail cl-route-check.

In addition, cl-route-check is unreliable in the operating mode where multiple front panel ports share a MAC address.
CumulusLinux-1.5.0-Final CumulusLinux-1.5.1 Tools

Known Issues in Cumulus Linux 1.5.1

Issues are categorized for easy review. Some issues are fixed but will be available in a later release. Future fixed issues are noted in the "fixed release"  column with the branch name the fix will be available in.

If the "fixed release" is "mainline", this means the fix is in Cumulus Linux's internal mainline branch, but not yet allocated to a customer branch/release.

 

Key Summary Description affectedTag Feature Category
RN-1 Restarting switchd flaps all switch ports switchd is a user-level process created by Cumulus Networks to provide an abstraction of the physical ports and the functionality provided by the switching ASIC SDK. switchd maps physical ports on the switching ASIC to logical ports (tap ports) in the kernel and ensures that CPU-bound packets are properly exposed on the proper logical objects to user level processes.

These exposed tap ports in the kernel are considered "running" if their file descriptors are open. If switchd exists, it closes the tap FDS, hence resulting in all links going down.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Hardware Abstraction Layer
RN-32 Adding bridges increases bootup time If the "bridge_maxwait" parameter is not set, the system will take approximately twice as long to bring the system up.

You should set the "bridge_maxwait" to 1.

Example config:

auto br1004
iface br1004 inet static
address 14.0.0.37
netmask 255.255.0.0
bridge_ports regex (swp[1|6|7|8].1004)
bridge_stp on
bridge_bridgeprio 32768
bridge_maxwait 1
bridge_ageing 200
bridge_fd 30
down ip addr flush dev br1004
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 2 (Control Protocols)
RN-12 The switch's forwarding of VLAN-tagged packets is different from Linux In Cumulus Linux, tagged packets sent to an untagged port get dropped. This is similar to general switch functionality from most vendors. However, in Linux, if a tagged packet is sent to an untagged port, it gets forwarded. CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 2 (Segmentation)
RN-71 SPAN: Cannot match an output L3 sub-interface Mirroring packets from one switch port to another works:

-A FORWARD --out-interface swp1 -j SPAN --dport swp5

As does mirroring inbound packets on a VLAN to a switch port:

-A FORWARD --in-interface swp1.100 -j SPAN --dport swp5

However, mirroring outbound packets from a VLAN to a switch port does not work:

-A FORWARD --out-interface swp1.100 -j SPAN --dport swp5

In summary:
* Matching output L3 main interface works fine.
* Matching input L3 sub-interface works fine.
* Matching output L3 sub-interface does not work.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 2 (Segmentation)
RN-80 Old link local IPv6 and MAC address are still used after the MAC address for a bridge changes Old MAC addresses for a bridge using the associated link local IPv6 address are used in the L2 and L3 headers in a router advertisement after the port is removed from the bridge.

To reproduce this:
1. Configure a bridge, br0, with swp1-4 in it.
2. Remove swp1 from br0 with the command "brctl delif br0 swp1"
3. Run a TCPDump session on swp2.

The TCPDump session reveals that the old MAC address, swp1's MAC address, and link local address are still being advertised. Here is a packet from the capture:

19:41:15.041412 44:38:39:00:01:81 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::4638:39ff:fe00:181 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 24
hop limit 64, Flags [none], pref medium, router lifetime 15s, reachable time 0s, retrans time 0s
source link-address option (1), length 8 (1): 44:38:39:00:01:82
0x0000: 4438 3900 0182

MAC 44:38:39:00:01:81 was the old MAC address for bridge br0 that was associated with swp1.
fe80::4638:39ff:fe00:181 is the link local address for swp1.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 2 (Segmentation)
RN-47 ip route append not supported, causing no support for unnumbered interfaces If the ip route append is used, routes will be appended into the kernel's ip table, but these routes will not show up in the ASIC.

ip route add and ip route delete are supported.

As a subsequent issue, this also means no support for unnumbered interfaces.

In essence, multiple routes in kernel for an interface will not show up in the ASIC; only the last route appears.

This affects unnumbered interfaces.

root@switch:~# ifconfig swp7.1007
swp7.1007 Link encap:Ethernet HWaddr 44:38:39:00:27:a5
inet addr:11.0.2.115 Bcast:11.0.2.127 Mask:255.255.255.240
inet6 addr: fe80::4638:39ff:fe00:27a5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:90 (90.0 B)

root@switch:~# ifconfig swp8.1007
swp8.1007 Link encap:Ethernet HWaddr 44:38:39:00:27:a6
inet addr:11.0.2.115 Bcast:11.0.2.127 Mask:255.255.255.240
inet6 addr: fe80::4638:39ff:fe00:27a6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:90 (90.0 B)

root@switch:~# ip route show table all | grep \.115
11.0.2.112/28 dev swp7.1007 proto kernel scope link src 11.0.2.115
11.0.2.112/28 dev swp8.1007 proto kernel scope link src 11.0.2.115

CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-56 ipv4/ipv6 forwarding disabled mode not recognized If either of the following is configured:
net.ipv4.ip_forward == 0
or:
net.ipv6.conf.all.forwarding == 0

The hardware still forwards packets if there is a neighbor table entry pointing to the destination.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-58 IPv6 route is installed and active in the routing table when the associated interface is down If an IPv6 address is assigned to a "down" interface, the associated route is still installed into the route table.

Also, the type of IPv6 address doesn't matter. Link local, site local, and global all exhibit the same problem.

If the interface is bounced up and down, then the routes are no longer in the route table.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-68 Blackhole/Unreachable/Prohibit route addition in IPv6 returns corresponding error codes IPv6 route operations indicate the destination action via returned error codes. In the example shown below where an unreachable route is being added, the return code is:

#define ENETUNREACH 101 /* Network is unreachable */


ip addr add 9000:1000:1000:1000::1/80 dev lo
root@switch$ sudo ip -6 route
unreachable 9000:1000:1000:1000::/80 dev lo proto kernel metric 256 error -101
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-77 New routes/ECMPs can evict existing/installed Cumulus Linux syncs routes between the kernel and the switching silicon. If the required resource pools in hardware fill up, new kernel routes can cause existing routes to move from being fully allocated to being partially allocated.

In order to avoid this, routes in the hardware should be monitored and kept below the ASIC limits.

For example, on systems with Trident+, the limits are as follows:

routes: 16384 <<<< if all routes are ipv4
long mask routes 256 <<<< i.e., routes with a mask longer than the route mask limit
route mask limit 64
host_routes: 8192
ecmp_nhs: 4044
ecmp_nhs_per_route: 52

That translates to about 77 routes with ECMP NHs, if every route has the maximum ECMP NHs.

Monitoring this in Cumulus Linux is performed via the cl-resource-query command:

root@switch:~# cl-resource-query
hosts : 3
all routes : 29
IP4 routes : 17
IP6 routes : 12
nexthops : 3
ecmp_groups : 0
ecmp_nexthops : 0
mac entries : 0 / 131072
bpdu entries : 500 / 512

The resource to monitor is the ecmp_nexthops. If this count is close to 4044, new ECMPs may evict existing routes.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Forwarding)
RN-41 Route summarization acts differently in OSPFv3 vs. OSPFv2 In OSPFv2, regardless of the order of the routes, the longer prefix is always chosen:

area 0.0.0.0 range 11.0.0.0/16
area 0.0.0.0 range 11.0.0.0/8

In this case, 11.0.0.0/16 is always used regardless of the input order.

In OSPFv3, this is not always the case. Different results occur depending on the order routes are entered.

In the following case we get the longest match:

area 0.0.0.0 range 2000:1000::/32
area 0.0.0.0 range 2000::/16

However, if we do it in the reverse order of:

area 0.0.0.0 range 2000::/16
area 0.0.0.0 range 2000:1000::/32

We get both summarizations.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-52 Parameters like the router ID and DR priority cannot be changed while OSPFv2/v3 is running Router ID and DR priority can only be changed by shutting down OSPFv2/v3, changing the ID, and restarting the OSPF process.

A change to the DR priority may not properly be reflected in the LSAs that are still aging out.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-53 In OSPFv3, a new router LSA may not be generated if the neighbor advertises a router LSA with a different interface ID If the neighboring router has the interface ID change in the Hello message, Cumulus Linux (Quagga) will not generate an update LSA. CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-61 BGP4 notifications missing for several conditions In certain conditions, Quagga bgpd silently closes the peering without sending a notification. For example, if BGP receives a message with an invalid message type or invalid message length.

Ideally on any one of these cases, bgpd should send out a notification message to the peer.

General functionality of BGP4 is not affected.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-62 Attributes of a BGP aggregate route may not be RFC-compliant When BGP is configured with an aggregate route and there are more specific routes of that aggregate, the BGP speaker needs to analyze the attributes of those specific routes while forming the attributes of the aggregate route. The corresponding rules are defined in RFC 4271, Sect. 9.2.2.2.

In certain cases, the user may observe non-compliant attribute formation for the aggregate route: for example, incorrect MED and ORIGIN attributes.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-63 BGP4 recursive route not supported Quagga's bpgd does not support recursive routing when it's resolving BGP routes. It does work with respect to IGP routes.
As an example:
A BGP route of the form
R1 -> N1
where the best match for N1 is also a BGP route:
N1/24 -> N2
(and N2 gets resolved through IGP or directly connected) -
will not get resolved.

Quagga's Zebra specifically does not allow for such types of routes. The Zebra route path resolution algorithm needs to be fixed.


Although, Cumulus Linux's version of Quagga -- 0.99.21 -- has errors in recursive routing, the community is actively trying to fix it and patches are being tested in 0.99.22.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-64 Configuring route-reflector-client requires specific order In configuring a route to be a route reflector client, the Quagga configuration must be specified in a specific order; otherwise, the router will not be a route reflector client.

The "neighbor <IPv4/IPV6> route-reflector-client" command must be done after the "neighbor <IPV4/IPV6> Activate" command; otherwise, the route-reflector-client command is ignored.

Sample configuration:

router bgp 65000
bgp router-id 0.0.0.4
bgp log-neighbor-changes
no bgp default ipv4-unicast
bgp cluster-id 0.0.0.4
bgp bestpath as-path multipath-relax
redistribute connected
neighbor 14.0.0.1 remote-as 65000
neighbor 14.0.0.1 route-reflector-client
neighbor 14.0.0.1 activate
neighbor 14.0.0.1 next-hop-self
neighbor 14.0.0.9 remote-as 65000
neighbor 14.0.0.9 activate
neighbor 14.0.0.9 next-hop-self
neighbor 2001:ded:beef::1 remote-as 65000
neighbor 2001:ded:beef:2::1 remote-as 65000
maximum-paths 4
maximum-paths ibgp 4
!
address-family ipv6
redistribute connected
neighbor 2001:ded:beef::1 activate
neighbor 2001:ded:beef::1 next-hop-self
neighbor 2001:ded:beef:2::1 route-reflector-client
neighbor 2001:ded:beef:2::1 activate
neighbor 2001:ded:beef:2::1 next-hop-self

maximum-paths 4
maximum-paths ibgp 4
exit-address-family

At runtime:
~~~~~~~~~

sp1# show ip bgp neighbor 14.0.0.1
BGP neighbor is 14.0.0.1, remote AS 65000, local AS 65000, internal link
BGP version 4, remote router ID 0.0.0.6
BGP state = Established, up for 00:23:49
Last read 23:31:36, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 2 0
Notifications: 0 0
Updates: 1 1
Keepalives: 25 24
Route Refresh: 0 0
Capability: 0 0
Total: 28 25
Minimum time between advertisement runs is 5 seconds

For address family: IPv4 Unicast
>>>>>>>>>>>>>>>>>>>>>> ROUTE REFLECTOR CLIENT NOT DISPLAYED
NEXT_HOP is always this router
Community attribute sent to this neighbor(both)
6 accepted prefixes

Connections established 1; dropped 0
Last reset never
Local host: 14.0.0.2, Local port: 179
Foreign host: 14.0.0.1, Foreign port: 40290
Nexthop: 14.0.0.2
Nexthop global: 2001:ded:beef::2
Nexthop local: fe80::202:ff:fe00:4
BGP connection: non shared network
Read thread: on Write thread: off

sp1#


Workaround:
~~~~~~~~~~

Define in following order
address-family ipv4 unicast
neighbor 14.0.0.9 activate
neighbor 14.0.0.9 next-hop-self
neighbor 14.0.0.9 route-reflector-client >>> Must be after Activate
exit-address-family
neighbor 2001:ded:beef:2::1 remote-as 65000
address-family ipv6 unicast
redistribute connected
maximum-paths 4
maximum-paths ibgp 4
neighbor 2001:ded:beef:2::1 activate
neighbor 2001:ded:beef:2::1 next-hop-self
neighbor 2001:ded:beef:2::1 route-reflector-client >>> Must be after activate
exit-address-family

Runtime status after change:
sp1# show ip bgp neighbors 14.0.0.9
BGP neighbor is 14.0.0.9, remote AS 65000, local AS 65000, internal link
BGP version 4, remote router ID 0.0.0.7
BGP state = Established, up for 00:13:59
Last read 22:35:13, hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
Route refresh: advertised and received(old & new)
Address family IPv4 Unicast: advertised and received
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 2 1
Keepalives: 15 14
Route Refresh: 0 0
Capability: 0 0
Total: 18 16
Minimum time between advertisement runs is 5 seconds

For address family: IPv4 Unicast
Route-Reflector Client >>>>>>>>>> PLEASE NOTE ME
NEXT_HOP is always this router
Community attribute sent to this neighbor(both)
6 accepted prefixes

Connections established 1; dropped 0
Last reset never
Local host: 14.0.0.10, Local port: 38813
Foreign host: 14.0.0.9, Foreign port: 179
Nexthop: 14.0.0.10
Nexthop global: 2001:ded:beef:2::2
Nexthop local: fe80::202:ff:fe00:6
BGP connection: non shared network
Read thread: on Write thread: off

sp1#
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-65 Virtual links in Quagga's OSPFv2 are non-operational Cumulus Networks testing has identified too many issues with virtual link support in Quagga's OSPFv2. The feature is unsupported. CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-74 BGP: TTL security hops requires ACL per peer to drop expired TTL BGP packers When configuring TTL security, rules must be put in to drop packets with exceeded TTL on BGL peers. This ensures that when the peers are taken down, the packets are properly discarded.

Here is an example of a rule to put in iptables:

root@switch:~# cat /etc/cumulus/acl/policy.d/bgp.rules
INGRESS_INTF = swp+

[iptables]

-A INPUT --in-interface $INGRESS_INTF -p tcp --dport bgp --src 11.0.0.6 -m ttl --ttl-eq 253 -j POLICE --set-mode pkt --set-rate 2000 --set-burst 1000 --set-class 7
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Layer 3 (Routing)
RN-87 Ignore Errors in switchd Log if One or More Interfaces Go Down during ARP Request

If the system sends ARP requests to resolve next hop information, and an interface goes down in the process, switchd will not recognize the outage, and an error message will be written to the switchd log. As a result, the routes that depend on the neighbor table entries will not get installed. However, the error message is benign.

At some point later, if switchd encounters a next hop where the IP address is not in the neighbor table, ARPs are sent out in processing the route's next hop list and an appropriate ARP response would result in the neighbor table getting filled for that IP address with the neighbor's MAC address. The route can now installed in hardware.

The following is an example of the error that appears in the switchd log:

1377281875.125649 2013-08-23 18:17:55 sync.c:3091 Bridge Macs Summary : 0 Added, 0 Deleted, 7 Updated in 193890 usecs
1377281885.123837 2013-08-23 18:18:05 sync.c:3091 Bridge Macs Summary : 0 Added, 0 Deleted, 7 Updated in 191843 usecs
1377281896.906472 2013-08-23 18:18:16 sync.c:2113 Route Summary : 0 Added, 0 Deleted, 5001 Updated in 2380987 usecs
1377281896.906566 2013-08-23 18:18:16 sync.c:2116 Neighbor Summary : 0 Added, 0 Deleted, 0 Updated in 2381050 usecs
1377281897.623128 2013-08-23 18:18:17 sync.c:3091 Bridge Macs Summary : 0 Added, 0 Deleted, 7 Updated in 289157 usecs
1377281900.763041 2013-08-23 18:18:20 sync.c:2523 Bridge Interface Summary : 0 Added, 0 Deleted, 1 Updated in 6949 usecs
1377281900.769285 2013-08-23 18:18:20 sync.c:2523 Bridge Interface Summary : 0 Added, 0 Deleted, 1 Updated in 5849 usecs
1377281900.776102 2013-08-23 18:18:20 sync.c:2523 Bridge Interface Summary : 0 Added, 0 Deleted, 2 Updated in 6458 usecs
1377281900.849408 2013-08-23 18:18:20 pkt_inj.c:350 ERR NHS probe for 34.0.0.22 on dev br3 failed : Network is unreachableRoute Summary : 0 Added, 5 Deleted, 5001 Updated in 2256289 usecs
1377281903.032513 2013-08-23 18:18:23 sync.c:2116 Neighbor Summary : 0 Added, 1 Deleted, 0 Updated in 2256335 usecs
1377281903.262519 2013-08-23 18:18:23 sync.c:2523 Bridge Interface Summary : 0 Added, 5 Deleted, 0 Updated in 74117 usecs
1377281903.513022 2013-08-23 18:18:23 sync.c:3091 Bridge Macs Summary : 0 Added, 2 Deleted, 4 Updated in 250285 usecs
1377281903.709135 2013-08-23 18:18:23 sync.c:3091 Bridge Macs Summary : 0 Added, 2 Deleted, 1 Updated in 195696 usecs
1377281904.445373 2013-08-23 18:18:24 sync.c:3091 Bridge Macs Summary : 2 Added, 0 Deleted, 2 Updated in 191687 usecs
1377281907.517178 2013-08-23 18:18:27 sync.c:3091 Bridge Macs Summary : 0 Added, 0 Deleted, 5 Updated in 193727 usecs
1377281910.098705 2013-08-23 18:18:30 sync.c:2113 Route Summary : 0 Added, 5001 Deleted, 0 Updated in 807673 usecs
1377281910.098787 2013-08-23 18:18:30 sync.c:2116 Neighbor Summary : 0 Added, 0 Deleted, 0 Updated in 807722 usecs
1377281917.536064 2013-08-23 18:18:37 sync.c:3091 Bridge Macs Summary : 0 Added, 0 Deleted, 6 Updated in 212500 usecs
1377281924.667599 2013-08-23 18:18:44 sync.c:2113 Route Summary : 5001 Added, 0 Deleted, 0 Updated in 1412682 usecs
1377281924.667683 2013-08-23 18:18:44 sync.c:2116 Neighbor Summary : 0 Added, 0 Deleted, 0 Updated in 1412731 usecs

CumulusLinux-1.5.1 Layer 3 (Routing)
RN-4 ifup/ifdown must be used for interfaces with IPv6 addresses defined in /etc/network/interfaces, otherwise the IPv6 interface will go down Two scenarios are shown below; one with ifup/ifdown, the other with ifconfig down.

With ifup/ifdown:

swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4231 errors:0 dropped:0 overruns:0 frame:0
TX packets:4342 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:412115 (402.4 KiB) TX bytes:425688 (415.7 KiB)

root@switch$ ifdown swp1
root@switch$ ifconfig swp1
swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:4248 errors:0 dropped:0 overruns:0 frame:0
TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB)

root@switch$ ifconfig swp1
swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:4248 errors:0 dropped:0 overruns:0 frame:0
TX packets:4356 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:413990 (404.2 KiB) TX bytes:427074 (417.0 KiB)

root@switch$ ifup swp1
ADDRCONF(NETDEV_UP): swp1: link is not ready
root@switch$ ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: link becomes ready

swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4250 errors:0 dropped:0 overruns:0 frame:0
TX packets:4362 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:414178 (404.4 KiB) TX bytes:427610 (417.5 KiB)

root@switch$



With ifconfig down:

ifconfig swp1
swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
inet6 addr: fec0:1000:1000:1000::2/10 Scope:Site
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:98 errors:0 dropped:0 overruns:0 frame:0
TX packets:111 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:13310 (12.9 KiB) TX bytes:12786 (12.4 KiB)

root@switch$ ifconfig swp1 down
root@switch$ ifconfig swp1
swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:126 errors:0 dropped:0 overruns:0 frame:0
TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:16998 (16.5 KiB) TX bytes:15998 (15.6 KiB)

root@switch$ ifconfig swp1 up
ADDRCONF(NETDEV_UP): swp1: link is not ready
root@switch$ ifconfig swp1ADDRCONF(NETDEV_CHANGE): swp1: link becomes ready

swp1 Link encap:Ethernet HWaddr 44:38:39:00:01:81
inet addr:11.0.0.2 Bcast:11.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::4638:39ff:fe00:181/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:130 errors:0 dropped:0 overruns:0 frame:0
TX packets:149 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:17474 (17.0 KiB) TX bytes:17154 (16.7 KiB)

CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Management Interface
RN-48 Agema 48x10GE switch eth0 driver reports eth0 as running even when PHY link is down The Agema 48x10GE eth0 driver reports eth0 as running even when the PHY link is down.

This can be really misleading in trying to diagnose a link-down situation on eth0. ethtool eth0 shows the correct PHY link status, but ifconfig shows eth0 as running, regardless of the PHY link status.

A fix will be released after Cumulus Linux 1.5.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Management Interface
RN-86 snmpwalk of ip-forward-mib Can Cause High CPU Utilization

ip-forward-mib is disabled in snmpd due to the poor performance of snmpd while handling many (1000+) forwarding table entries.

See these links for more information:

CumulusLinux-1.5.1 Monitoring
RN-88 SNMP Support for Quagga Is NOT Provided in Cumulus Linux

Cumulus Linux 1.5.1 does not provide SNMP support for Quagga.

Due to this change, you must remove the line listed in each of the following Quagga configuration files in /etc/quagga:

  • In bgpd.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.2 quagga_bgpd
  • In ospf6d.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.6 quagga_ospf6d
  • In ospfd.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.5 quagga_ospfd
  • In zebra.conf, remove this line: 
    smux peer 1.3.6.1.4.1.3317.1.2.1 quagga_zebra

If these lines are not removed, the Quagga daemons listed here will not start.

CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Monitoring
RN-69 Statistics for ACL rules are reset to zero whenever the ACL rules are installed The installation of ACL rules in hardware (using cl-acltool) is done atomically using a ping pong buffer scheme. Thus, old counters are lost anytime a rule is changed (added or deleted). Thus, ACL stats for all the rules get zeroed out, irrespective of whether the rules file installation succeeds or fails.

root@switch:~# iptables -L -v
Chain INPUT (policy ACCEPT 291 packets, 20717 bytes)
pkts bytes target prot opt in out source destination
300 134K DROP udp -- swp+ any 11.0.0.1 11.0.0.10 udp spt:800 dpt:1000
0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:3
1 102 POLICE icmp -- swp+ any anywhere anywhere POLICE class:2 rate:1 burst:4
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootps SETCLASS class:2
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootpc SETCLASS class:2
2 892 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type IPROUTER POLICE class:2 rate:1 burst:4

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 242 packets, 37343 bytes)
pkts bytes target prot opt in out source destination



root@switch:~# cl-acltool -i -p 00control_plane.rules
Using user provided rule file 00control_plane.rules
Reading rule file 00control_plane.rules ...
Processing rules in file 00control_plane.rules ...
Installing acl policy... done.



root@switch:~# iptables -L -v
Chain INPUT (policy ACCEPT 8 packets, 560 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP udp -- swp+ any 11.0.0.1 11.0.0.10 udp spt:800 dpt:1000
0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:3
0 0 POLICE icmp -- swp+ any anywhere anywhere POLICE class:2 rate:1 burst:4
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootps SETCLASS class:2
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:bootpc SETCLASS class:2
0 0 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type IPROUTER POLICE class:2 rate:1 burst:4

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 5 packets, 612 bytes)
pkts bytes target prot opt in out source destination
root@switch:~#

CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Security
RN-70 ACL: Bridge traffic that matches a LOG ACTION rule is not logged in syslog For example, a bridge with switch ports swp1, swp2, swp3 as bridge members is configured. ACL rules to LOG and DROP for icmp traffic are configured.

Ping requests are sent from host1 on swp1 to host3 on swp3, and the following was observed:
* Counters for both LOG and DROP ACL rules are incrementing properly, but the packets are not showing up on /var/log/syslog.
* Packets that are copied to the CPU from hardware for the LOG rule are dropped due to the check in kernel to disable software bridging for hardware bridged packets.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Security
RN-81 Hardware enforcement for bridging packets that match both ebtables and iptables/ip6tables rules have different behavior from Linux kernel enforcement iptables Issue:
The behavior for bridging packets that match the rules for both ebtables and iptables is different from netfilter enforcement for packets matching these tables in the Linux kernel.

ebtables are always enforced before iptables for bridging packets, whereas iptables are enforced for hardware.

ip6tables Issue:
The behavior for bridging packets that match the rules for both ebtables as well as ip6tables is different from netfilter enforcement for packets matching these tables in the Linux kernel.

ebtables are always enforced before ip6tables for bridging packets in the Linux kernel, whereas the actions in both these tables (ebtables/ip6tables) for the packet will try to be enforced.
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Security
RN-10 cl-phy-update doesn't support aggregated ports Ports can be aggregated into a larger interface in Cumulus Linux. Unfortunately support for aggregated ports is not yet supported when running cl-phy-update.

If there are any ganged ports during a SW upgrade it is recommended to ungang these ports
CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Tools
RN-85 Error log messages displayed with cl-resource-query when following a switchd restart If you use cl-resource-query following a switchd restart, a transient hardware error may occur and trigger several error messages. Based on current analysis, the error is harmless, and Cumulus Networks will work to remove it in a future release. CumulusLinux-1.5.0-Final, CumulusLinux-1.5.1 Tools
RN-313 High memory utilization by snmpd following MIB walks

When performing an MIB walk on Cumulus Linux running the standard NET-SNMP 5.4.3 package, memory is consumed and not returned. Over time, this can lead to sub-optimal performance and ultimately can cause the system to hang and require a reboot.

Monitor snmpd memory utilization using ps aux and system memory utilization with free or cat /proc/meminfo. If required, restart snmpd using service snmpd restart.


RN-372 (CM-9360)
Security Update for CVE-2015-7547: glibc getaddrinfo Stack-based Buffer Overflow Vulnerability For details on this issue and how to upgrade, read this article.
Have more questions? Submit a request

Comments

Powered by Zendesk