Issue

Logging is useful for determining:

• Admin state (whether the port/bridge/bond up or down)
• Physical state (whether the connection for the port/bridge/bond up or down)

... and the current level is not enough in /var/log/.

Environment

All versions of Cumulus Linux.

Resolution

• Utilizing ip monitor
• Running in the background
• Automatically logging on boot
• Looking at the log output
• Admin/physical state troubleshooting table

Utilizing ip monitor

Use the Linux ip monitor tool to monitor link state.

[email protected]:~$ ip -timestamp monitor link

• The -timestamp command allows us to timestamp each message
• The 'link' keyword specifies to only pay attention to link changes

Running in the Background

What if I want to run it in the background and redirect it to a log?

Cumulus Linux is Linux so use the & and > to redirect the output:

[email protected]:~$ sudo sh -c "ip -timestamp monitor link > /var/log/link.log" &

The redirect (>) in the previous example requires you to use the quotation marks around the two commands subjected to the redirection clause (>) so that both commands are run as the root user. If using the root account (disabled by default on Cumulus Linux 2.0 and later) there is no need:

[email protected]:~# ip -timestamp monitor link > /var/log/link.log &

Automatically Logging on Boot

What if I automatically want to log link activity, even if the switch is rebooted?

Utilize the rc.local startup script located at /etc/rc.local. Here rc.local is already configured on a Cumulus Linux switch:

    
    [email protected]:~$ cat /etc/rc.local
    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will "exit 0" on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    #
    # By default this script does nothing.
    ip -timestamp monitor link > /var/log/link.log
    
    exit 0
    
    [email protected]:~$
    

Looking at the Log Output

What does admin state vs. physical state look like in the log?

Let's go ahead and turn a port off and on and look at the log.

[email protected]:~$ sudo ip link set swp1 down

[email protected]:~$ sudo ip link set swp1 up

[email protected]:~$ cat /var/log/link.log
Timestamp: Wed Jul 16 19:00:35 2014 350805 usec
3: swp1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond24 state DOWN
    link/ether 44:38:39:00:25:d9 brd ff:ff:ff:ff:ff:ff
Timestamp: Wed Jul 16 19:00:37 2014 840831 usec
3: swp1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond24 state DOWN
    link/ether 44:38:39:00:25:d9 brd ff:ff:ff:ff:ff:ff

/var/log/linkstate

Cumulus Linux provides a link state log, that outputs to /var/log/linkstate, and provides concise records of all physical and logical network link state changes. In particular, the output shows up/down changes, for both physical and admin links.

This log has been added as link state changes occur far less often than other logged events, and a separate log ensures link state changes are recorded for a longer period of time, as the information will not be rotated out with the rest of syslog.

[email protected]:mgmt-vrf:~$ sudo tail /var/log/linkstate
2017-05-22T11:21:07.085595-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin down
2017-05-22T11:21:07.086167-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper down
2017-05-22T11:21:07.153894-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: down
2017-05-22T11:23:02.968972-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin up
2017-05-22T11:23:05.152445-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: up
2017-05-22T11:23:05.165747-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper up
2017-05-22T11:23:06.683631-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin down
2017-05-22T11:23:06.684531-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper down
2017-05-22T11:23:06.723817-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: down

Admin/Physical State Troubleshooting Table

Now let's look at every possible scenario to make troubleshooting easier.

swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500

swp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500

swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500

swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500

Warning Notice how Admin down and down/down look identical.

For additional troubleshooting of physical ports, read the documentation on duplex, speed and auto-negotiation settings.