This knowledge base has moved to the documentation site. Please visit the knowledge base here for the most up to date content. This site is no longer maintained.

Using dtach for In-band apt-get Upgrades

Follow

Issue

My ssh session was terminated while performing an in-band upgrade.

Environment

  • Cumulus Linux 2.5.x, 3.x
  • In-band ssh connection: upgrading via ssh connection over a logical (bridge or bond) or switch (swp) port, rather than the management eth0 port.

Resolution

When upgrading in-band, it is recommended to run the upgrade with the dtach or screen command. This makes it possible to re-attach to the upgrade process in the event the connection is interrupted.

Using dtach 

For Cumulus Linux 3.x:

cumulus@switch$ sudo apt-get update
cumulus@switch$ dtach -c /tmp/apt-get -z sudo apt-get upgrade

For Cumulus Linux 2.5.x:

cumulus@switch$ sudo apt-get update
cumulus@switch$ dtach -c /tmp/apt-get -z sudo apt-get dist-upgrade

Where:

  • /tmp/apt-get is a socket that will be used to re-attach later
  • -z disables the suspend key from suspending dtach

To re-attach to the apt-get process after reconnecting to the switch:

cumulus@switch$ dtach -a /tmp/apt-get

If the dtach socket file does not exist as in the following error:

cumulus@switch$ dtach -a /tmp/apt-get
dtach: /tmp/apt-get: No such file or directory

... then upgrade has finished. Prior to rebooting (if needed) results of the upgrade can be reviewed in /var/log/apt/term.log.

cumulus@switch$ less /var/log/apt/term.log

Using screen

Start a screen session specifying a name to re-attach to later:

screen -S apt-get

In the new screen session, proceed with the upgrade process.

For Cumulus Linux 3.x:

cumulus@switch$ sudo apt-get update
cumulus@switch$ sudo apt-get upgrade

For Cumulus Linux 2.5.x:

cumulus@switch$ sudo apt-get update
cumulus@switch$ sudo apt-get dist-upgrade

To re-attach to the apt-get process after reconnecting to the switch:

screen -r apt-get

Root Cause

Upgrades involving certain packages, such as switchd or FRR/Quagga, may result in a temporary loss of in-band ssh connections (generally a minute or two) to the switch.

Comments

This support portal has moved

Cumulus Networks is now part of the NVIDIA Networking Business Unit! The NVIDIA Cumulus Global Support Services (GSS) team has merged its operations with the NVIDIA Mellanox support services team.

You can access NVIDIA Cumulus support content from the Mellanox support portal.

You open and update new cases on the Mellanox support portal. Any previous cases that have been closed have been migrated to the Mellanox support portal.

Cases that are still open on the Cumulus portal will continue to be managed on the Cumulus portal. Once these cases close, they will be moved to the Mellanox support portal.

Powered by Zendesk