By default, the out-of-band management interface (eth0) on a Cumulus Linux system shares a common routing table with the in-band (swp) ports of the system. This can cause traffic to take an undesired path or allow in-band traffic to try and route via the eth0 interface.
This article provides examples for setting up an alternative route table, IP rules and iptables configuration to enforce management traffic to utilize the eth0 interface. Since each site is different, these solutions should be used as a suggestion and modified to meet your site's specific needs.
- Cumulus Linux, all versions
- Static IP address assigned on the eth0 interface
The solution involves three parts:
- Configuring eth0 to run multiple commands automatically
- Enabling iptables traffic selection
- Securing eth0 from outside traffic
Configuring eth0 to Automatically Run Multiple Commands
/etc/network/interfaces file, find the section that configures the eth0 interface and replace it with the following lines.
iface eth0 inet static
address <IP Address>/<Mask>
up ip route add default via <Gateway> dev eth0 tab 1
down ip route delete default via <Gateway> dev eth0 tab 1
up ip rule add from <IP Address>/32 tab 1
down ip rule delete from <IP Address>/32 tab 1
up ip rule add fwmark 1 table 1
down ip rule delete fwmark 1 table 1
up iptables -A POSTROUTING -t nat -o eth0 -j SNAT --to <IP Address>
The above code sets up route table 1 which has a default route pointed at the <Gateway> address. If you have more routes that you want to manually add, they can be added in a similar way. It then adds a rule which will say that if any traffic is coming from the IP address of the eth0 interface, to lookup in table 1, not in the default table. These rules alone will cause any traffic using the IP address of eth0 to follow the table 1 routing table. For any inbound sessions (like SNMP or SSH), this is all that is needed.
For outbound traffic, the remaining lines are required. The mark rule is used later to tie outbound traffic to looking up in table 1 and the last line sets up a rewrite rule to insure that traffic going out of eth0 comes from the IP address of eth0.
iptables Traffic Selection
To insure that outbound traffic uses route table 1, the following commands need to be run.
/sbin/iptables -A OUTPUT -t mangle -d <My IP address range>/<Mask> -j ACCEPT
/sbin/iptables -A OUTPUT -t mangle -j MARK --set-mark 1
The first line tells the system to not-rewrite traffic that is going to the address space used by my hosts. If you have more then one range, simply add more lines like it. The second line causes any remaining traffic to be set with the mark of 1 which will be looked up in route table 1 due to the earlier rules.
It is important to note that the mangle table is utilized by the
cl-acltool command for QoS. Since the MARK command is not supported by the hardware,
cl-acltool will not allow you to add the above set of commands into the normal files located in
/etc/cumulus/acl/policy.d. Your version of the above commands must be re-run after each
cl-acltool -i to restore sending the outgoing management traffic via the eth0 interface, including at boot time once the system is fully up and running and acl's have been applied. One way to accomplish both is to create a script that you call in
rc.local or after using
For example, if you had the following script called
#!/bin/bash # Sleep if asked if [ ! "$1" = "" ] then sleep $1 fi /sbin/iptables -A OUTPUT -t mangle -d <My IP address range>/<Mask> -j ACCEPT /sbin/iptables -A OUTPUT -t mangle -j MARK --set-mark 1
And then add the following line to your
/usr/local/bin/iptablesfix 30 &
cl-acltool, you can then follow up with the same script:
cl-acltool -i && iptablesfix
To protect the system from hosts that might try and reach the eth0 IP address, a line like the following should be added to your
/etc/cumulus/acl/policy.d/99control_plane_catch_all.rules file, which causes this traffic to drop.
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -d <IP Address> -j DROP
- Mangle tables rules need to be reset each time
- Dynamic IP addresses can be supported by creating a DHCP client exit script that applies the IP rules shown in the first section of the solution. You can find a starter reference for developing this at Ubuntu Dual DHCP.