This knowledge base has moved to the documentation site. Please visit the knowledge base here for the most up to date content. This site is no longer maintained.

Monitoring Interface Administrative State and Physical State on Cumulus Linux

Follow

Issue

Logging is useful for determining:

  • Admin state (whether the port/bridge/bond up or down)
  • Physical state (whether the connection for the port/bridge/bond up or down)

... and the current level is not enough in /var/log/.

Environment

All versions of Cumulus Linux.

Resolution

Utilizing ip monitor

Use the Linux ip monitor tool to monitor link state.

cumulus@switch:~$ ip -timestamp monitor link
  • The -timestamp command allows us to timestamp each message
  • The 'link' keyword specifies to only pay attention to link changes

Running in the Background

What if I want to run it in the background and redirect it to a log?

Cumulus Linux is Linux so use the & and > to redirect the output:

cumulus@switch:~$ sudo sh -c "ip -timestamp monitor link > /var/log/link.log" &

The redirect (>) in the previous example requires you to use the quotation marks around the two commands subjected to the redirection clause (>) so that both commands are run as the root user. If using the root account (disabled by default on Cumulus Linux 2.0 and later) there is no need:

root@switch:~# ip -timestamp monitor link > /var/log/link.log &

Automatically Logging on Boot

What if I automatically want to log link activity, even if the switch is rebooted?

Utilize the rc.local startup script located at /etc/rc.local. Here rc.local is already configured on a Cumulus Linux switch:

    
    cumulus@switch:~$ cat /etc/rc.local
    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will "exit 0" on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    #
    # By default this script does nothing.
    ip -timestamp monitor link > /var/log/link.log
    
    exit 0
    
    cumulus@switch:~$
    

Looking at the Log Output

What does admin state vs. physical state look like in the log?

Let's go ahead and turn a port off and on and look at the log.

cumulus@switch:~$ sudo ip link set swp1 down

cumulus@switch:~$ sudo ip link set swp1 up

cumulus@switch:~$ cat /var/log/link.log
Timestamp: Wed Jul 16 19:00:35 2014 350805 usec
3: swp1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond24 state DOWN
    link/ether 44:38:39:00:25:d9 brd ff:ff:ff:ff:ff:ff
Timestamp: Wed Jul 16 19:00:37 2014 840831 usec
3: swp1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond24 state DOWN
    link/ether 44:38:39:00:25:d9 brd ff:ff:ff:ff:ff:ff

/var/log/linkstate

Cumulus Linux provides a link state log, that outputs to /var/log/linkstate, and provides concise records of all physical and logical network link state changes. In particular, the output shows up/down changes, for both physical and admin links.

This log has been added as link state changes occur far less often than other logged events, and a separate log ensures link state changes are recorded for a longer period of time, as the information will not be rotated out with the rest of syslog.

cumulus@leaf01:mgmt-vrf:~$ sudo tail /var/log/linkstate
2017-05-22T11:21:07.085595-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin down
2017-05-22T11:21:07.086167-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper down
2017-05-22T11:21:07.153894-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: down
2017-05-22T11:23:02.968972-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin up
2017-05-22T11:23:05.152445-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: up
2017-05-22T11:23:05.165747-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper up
2017-05-22T11:23:06.683631-04:00 leaf01 switchd[19517]: sync_base.c:596 swp51: ifindex 54, admin down
2017-05-22T11:23:06.684531-04:00 leaf01 switchd[19517]: sync_base.c:604 swp51: ifindex 54, oper down
2017-05-22T11:23:06.723817-04:00 leaf01 switchd[19517]: nic.c:223 nic_set_carrier: swp51: setting kernel carrier: down

Admin/Physical State Troubleshooting Table

Now let's look at every possible scenario to make troubleshooting easier.

Admin State Physical State ip link show Output
up up
swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500
up down
swp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500
down up
swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500
down down
swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500

Warning Notice how Admin down and down/down look identical.

For additional troubleshooting of physical ports, read the documentation on duplex, speed and auto-negotiation settings.

Comments

This support portal has moved

Cumulus Networks is now part of the NVIDIA Networking Business Unit! The NVIDIA Cumulus Global Support Services (GSS) team has merged its operations with the NVIDIA Mellanox support services team.

You can access NVIDIA Cumulus support content from the Mellanox support portal.

You open and update new cases on the Mellanox support portal. Any previous cases that have been closed have been migrated to the Mellanox support portal.

Cases that are still open on the Cumulus portal will continue to be managed on the Cumulus portal. Once these cases close, they will be moved to the Mellanox support portal.

Powered by Zendesk