This knowledge base has moved to the documentation site. Please visit the knowledge base here for the most up to date content. This site is no longer maintained.

Using netconsole with syslog on Cumulus Linux Switches

Follow

{{table_of_contents}}

About netconsole

netconsole is a feature of Linux that allows you to redirect kernel messages output from dmesg to a location across the network using UDP. These messages can be captured and stored on a syslog server for investigating issues on a Cumulus Linux switch where the dmesg output was generated. This is particularly useful in situations where a physical console is not connected and you need to debug kernel events such as system crashes and unexpected reboots.

netconsole is not a replacement for a physical console. It does not provide an interactive console to the switch; it is a remote logging service only. netconsole is also limited in that it is not available until the network has initialized on boot. Log data from early in the boot cycle does not get captured. That does not mean you should avoid using netconsole. It is a great tool to use whenever a physical console is not available to log data.

Configure the netconsole Module on Your Cumulus Linux Switch

Note: You must reboot the switch at the end of this process to apply the changes.

  1. Set up the netconsole kernel module to load on boot:
    $ echo netconsole | sudo tee /etc/modules-load.d/netconsole.conf
  2. Configure the netconsole kernel module options to point to your syslog server. The format for the options is as follows:
    netconsole=[+][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
       where
            +            if present, enable extended console support
            src-port     source for UDP packets (defaults to 6665)
            src-ip       source IP to use (interface address)
            dev          network interface (eth0)
            tgt-port     port for logging agent (6666)
            tgt-ip       IP address for logging agent
            tgt-macaddr   ethernet MAC address for logging agent (broadcast)
        

    Note: Some of these parameters are optional.

    This example command uses the following configuration:

    • The IP address of the management port (eth0) on the switch is 10.20.30.40.
    • The IP address of the destination (syslog) server is 10.20.30.255.
    • The MAC address of the destination (syslog) server is 00:22:33:aa:bb:cc. If the syslog server IP address is on a different subnet than the switch, specify the MAC address of the default gateway instead. 
    • The destination port for log traffic is port 6666.

    You use this configuration to create a module options file named /etc/modprobe.d/netconsole.conf:

    $ echo 'options netconsole netconsole=@10.20.30.40/eth0,6667@10.20.30.255/00:22:33:aa:bb:cc' | sudo tee /etc/modprobe.d/netconsole.conf
  3. You can increase or decrease the amount of data you want to log.
    • To increase the amount of data being logged by the kernel (see linuxconfig.org/introduction-to-the-linux-kernel-log-levels), adjust the log level. By default, a Cumulus Linux switch logs kernel data at level 3 (KERN_ERR). It may be useful to log all the data when trying to debug an issue. To do this, increase the kernel printk value to 7 in the /etc/systctl.d/99-sysctl.conf file:
      $ echo 'kernel.printk = 7 4 1 7' | sudo tee -a /etc/sysctl.d/99-sysctl.conf
    • To limit the data to just kernel panic logs, set the kernel module option oops_only to 1. You append oops_only=1 to the command you used in step 3 above.
      $ echo 'options netconsole netconsole=@10.90.2.25/eth0,6666@10.90.2.112/ac:1f:6b:49:af:f0 oops_only=1' | sudo tee /etc/modprobe.d/netconsole.conf
            
  4. Reboot the switch. The settings are applied during the boot sequence.

Create a Running Configuration on a Cumulus Linux Switch

The following procedure only impacts the running kernel on the switch; this is known as a non-persistent configuration. Once the switch reboots, these settings are lost.

Note: This procedure does not work if eth0 is in a management VRF. If you need to have eth0 in a management VRF, use the persistent configuration method above instead.

  1. Increase the kernel logging level (optional).
    $ sudo dmesg -n 7
  2. Load the netconsole kernel module with the appropriate options. The following command uses the same example configuration from above:

    $ sudo modprobe netconsole netconsole=@10.20.30.40/eth0,6666@10.20.30.255/00:22:33:aa:bb:cc

    To use the oops_only setting, load the netconsole kernel module with the appropriate options:

    $ sudo modprobe netconsole netconsole=@10.20.30.40/eth0,6666@10.20.30.255/00:22:33:aa:bb:cc oops_only=1

Configure an rsyslog Server to Receive the Console Log Data

The following steps show how to configure an rsyslog server to receive UDP traffic on port 6666 from 2 devices and create separate log files for each. You can add this to your existing rsyslog configuration. These steps must be performed by the root (super) user on your server.

  1. Create a specific configuration file with your favourite editor:
    # vi /etc/rsyslog.d/remote-netconsole.conf

    The file should contain:

    $ModLoad imudp
    $RuleSet remote
    
    # For each IP address that you want to store logs from,
    # add and modify the following two (!) lines:
    if $fromhost-ip=='10.20.30.40' then /var/log/remote/leafswitch1/console.log
    if $fromhost-ip=='10.20.30.41' then /var/log/remote/spineswitch2/console.log
    & stop
    
    $InputUDPServerBindRuleset remote
    $UDPServerRun 6666
    
    $RuleSet RSYSLOG_DefaultRuleset
        

    Note: The highlighted text should be changed to match the IP addresses of your switches and appropriate destination log files.

  2. Create a directory called /var/log/remote (or use a directory of your choice) to store the log files:
    # mkdir /var/log/remote
  3. Restart rsyslog.
    # systemctl restart rsyslog.service

Test the Setup by Intentionally Crashing a Switch

You can invoke a kernel panic to test the process.

Note: This causes a catastrophic failure of the switch and results in an immediate reboot. Please ensure your network is prepared for this to occur and you understand the consequences.

Log in to the switch you would like to crash and run the following command:

$ echo c | sudo tee /proc/sysrq-trigger

If the process is working correctly, you should see log data sent to the rsyslog server.

Log File Sample Output

Here is some sample output from the rsyslog server:

May 12 17:13:59 leafswitch1.network.com [17593.272492] sysrq: SysRq :
May 12 17:13:59 Trigger a crash
May 12 17:13:59 leafswitch1.network.com [17593.277181] BUG: unable to handle kernel
May 12 17:13:59 NULL pointer dereference
May 12 17:13:59 leafswitch1.network.com  at           (null)
May 12 17:13:59 leafswitch1.network.com [17593.285951] IP:
May 12 17:13:59 leafswitch1.network.com  [<ffffffff81496256>] sysrq_handle_crash+0x16/0x20
May 12 17:13:59 leafswitch1.network.com [17593.292773] PGD 4cb06067
May 12 17:13:59 PUD 4ca44067
May 12 17:13:59 PMD 0
May 12 17:13:59 leafswitch1.network.com
May 12 17:13:59 leafswitch1.network.com [17593.297566] Oops: 0002 [#1]
May 12 17:13:59 SMP
..

 

Comments

This support portal has moved

Cumulus Networks is now part of the NVIDIA Networking Business Unit! The NVIDIA Cumulus Global Support Services (GSS) team has merged its operations with the NVIDIA Mellanox support services team.

You can access NVIDIA Cumulus support content from the Mellanox support portal.

You open and update new cases on the Mellanox support portal. Any previous cases that have been closed have been migrated to the Mellanox support portal.

Cases that are still open on the Cumulus portal will continue to be managed on the Cumulus portal. Once these cases close, they will be moved to the Mellanox support portal.

Powered by Zendesk