Using IP Rules to Separate Out-of-band Management Traffic from In-band Traffic

Follow

Issue

By default, the out-of-band management interface (eth0) on a Cumulus Linux system shares a common routing table with the in-band (swp) ports of the system. This can cause traffic to take an undesired path or allow in-band traffic to try and route via the eth0 interface.  

This article provides examples for setting up an alternative route table, IP rules and iptables configuration to enforce management traffic to utilize the eth0 interface. Since each site is different, these solutions should be used as a suggestion and modified to meet your site's specific needs.

Environment

  • Cumulus Linux, all versions
  • Static IP address assigned on the eth0 interface

Solution

The solution involves three parts:

  • Configuring eth0 to run multiple commands automatically
  • Enabling iptables traffic selection
  • Securing eth0 from outside traffic

Configuring eth0 to Automatically Run Multiple Commands

In your /etc/network/interfaces file, find the section that configures the eth0 interface and replace it with the following lines.

auto eth0
iface eth0 inet static
address <IP Address>/<Mask>
up ip route add default via <Gateway> dev eth0 tab 1
down ip route delete default via <Gateway> dev eth0 tab 1
up ip rule add from <IP Address>/32 tab 1
down ip rule delete from <IP Address>/32 tab 1
up ip rule add fwmark 1 table 1
down ip rule delete fwmark 1 table 1
up iptables -A POSTROUTING -t nat -o eth0 -j SNAT --to <IP Address>

The above code sets up route table 1 which has a default route pointed at the <Gateway> address. If you have more routes that you want to manually add, they can be added in a similar way. It then adds a rule which will say that if any traffic is coming from the IP address of the eth0 interface, to lookup in table 1, not in the default table. These rules alone will cause any traffic using the IP address of eth0 to follow the table 1 routing table. For any inbound sessions (like SNMP or SSH), this is all that is needed.

For outbound traffic, the remaining lines are required. The mark rule is used later to tie outbound traffic to looking up in table 1 and the last line sets up a rewrite rule to insure that traffic going out of eth0 comes from the IP address of eth0.

iptables Traffic Selection

To insure that outbound traffic uses route table 1, the following commands need to be run.

/sbin/iptables -A OUTPUT -t mangle -d <My IP address range>/<Mask> -j ACCEPT
/sbin/iptables -A OUTPUT -t mangle -j MARK --set-mark 1

The first line tells the system to not-rewrite traffic that is going to the address space used by my hosts.  If you have more then one range, simply add more lines like it.  The second line causes any remaining traffic to be set with the mark of 1 which will be looked up in route table 1 due to the earlier rules.

It is important to note that the mangle table is utilized by the cl-acltool command for QoS. Since the MARK command is not supported by the hardware, cl-acltool will not allow you to add the above set of commands into the normal files located in /etc/cumulus/acl/policy.d. Your version of the above commands must be re-run after each cl-acltool -i to restore sending the outgoing management traffic via the eth0 interface, including at boot time once the system is fully up and running and acl's have been applied.  One way to accomplish both is to create a script that you call in rc.local or after using cl-acltool.

For example, if you had the following script called /usr/local/bin/iptablesfix:

#!/bin/bash
# Sleep if asked
if [ ! "$1" = "" ]
then
 sleep $1
fi
/sbin/iptables -A OUTPUT -t mangle -d <My IP address range>/<Mask> -j ACCEPT
/sbin/iptables -A OUTPUT -t mangle -j MARK --set-mark 1

And then add the following line to your rc.local:

/usr/local/bin/iptablesfix 30 &

After running cl-acltool, you can then follow up with the same script:

cl-acltool -i && iptablesfix

Security

To protect the system from hosts that might try and reach the eth0 IP address, a line like the following should be added to your /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules file, which causes this traffic to drop.

-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -d <IP Address> -j DROP

Caveats

  • Mangle tables rules need to be reset each time cl-acltool is run.

Notes

  • Dynamic IP addresses can be supported by creating a DHCP client exit script that applies the IP rules shown in the first section of the solution. You can find a starter reference for developing this at Ubuntu Dual DHCP.

 

Have more questions? Submit a request

Comments

Powered by Zendesk