[RETIRED]: Default ARP Settings in Cumulus Linux


Note: This article has been retired; please read the ARP chapter in the Cumulus Linux user guide

Cumulus Networks has implemented several changes from standard Debian Linux ARP behavior, since Cumulus Linux is an operating system for routers/switches rather than end-systems (servers). This article describes the differences in ARP behavior, why the changes were made, where the changes were implemented, and how to change port-specific values.

Standard Debian ARP Behavior and the Tunable ARP Parameters

Five tunable ARP parameters are available in Debian:

  • arp_accept
  • arp_announce
  • arp_filter
  • arp_ignore
  • arp_notify

These parameters are described in the Linux documentation, with snippets included in italics later in this article for each parameter description.

In a standard Debian installation, all of these ARP parameters are set to 0, leaving the router as wide open and unrestricted as possible. These settings are based on the assertion made long ago that Linux IP addresses are a property of the device, not a property of an individual interface. This means that an ARP request or reply could be sent on one interface containing an address residing on a different interface. While this unrestricted behavior makes sense for an end-system, it is not the normal behavior of a router. Routers expect the MAC/IP mappings supplied by ARP to match the physical topology, with the IP addresses matching the interfaces on which they reside. With these tunable ARP parameters, Cumulus Linux has been able to specify the behavior to match the expectations of a router.

ARP Tunable Parameter Settings in Cumulus Linux

As of Cumulus Linux 2.2.2, the ARP tunable parameters are set to the following values. Each parameter listed here is described in detail below, including why Cumulus Linux chose the value used.

Parameter Setting
arp_accept 0
arp_announce 2
arp_filter 0
arp_ignore 1
arp_notify 1


arp_accept - BOOLEAN

Define behavior for gratuitous ARP frames whose IP is not already present in the ARP table:

0 - Don't create new entries in the ARP table.

1 - Create new entries in the ARP table.

As of version 2.2.2, Cumulus Linux uses the default arp_accept behavior of not creating new entries in the ARP table when a gratuitous ARP is seen on an interface or when an ARP reply packet is received. Earlier versions of Cumulus Linux set this parameter to 1, populating the ARP table with entries when gratuitous ARPs or when ARP reply packets are seen on an interface, even if the interface was not operating in Layer 3. Also prior to version 2.2.2, the arp_accept parameter could not be specified on a per-port basis and could only be turned on or off for all ports on the switch.

Beginning with Cumulus Linux 2.2.2, however, an individual interface can have the arp_accept behavior set differently than the remainder of the switch if needed. A description on how to apply this port-specific behavior is in the last section of this article.

arp_announce - INTEGER

Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on interface:

0 - (default) Use any local address, configured on any interface.

1 - Try to avoid local addresses that are not in the target's subnet for this interface. This mode is useful when target hosts reachable via this interface require the source IP address in ARP requests to be part of their logical network configured on the receiving interface. When we generate the request we will check all our subnets that include the target IP and will preserve the source address if it is from such subnet. If there is no such subnet we select source address according to the rules for level 2.

2 - Always use the best local address for this target. In this mode we ignore the source address in the IP packet and try to select local address that we prefer for talks with the target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce.

The default Debian behavior with arp_announce set to 0 is to send gratuitous ARPs or ARP requests using any local source IP address, not limiting the IP source of the ARP packet to an address residing on the interface used to send the packet. This reflects the view historically held in Linux that IP addresses reside inside the device and are not considered a property of a specific interface.

Routers expect a different relationship between the IP address and the physical network. Adjoining routers will be looking for MAC/IP addresses to reach a next-hop residing on a connecting interface for transiting traffic. By setting the arp_announce parameter to 2, Cumulus Linux uses the best local address for each ARP request, preferring primary addresses on the interface used to send the ARP. This most closely matches traditional router ARP request behavior.

arp_filter - BOOLEAN

0 - (default) The kernel can respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. Only for more complex setups like load- balancing, does this behavior cause problems.

1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface (therefore you must use source based routing for this to work). In other words it allows control of which cards (usually 1) will respond to an ARP request.

arp_filter for the interface will be enabled if at least one of conf/{all,interface}/arp_filter is set to TRUE, it will be disabled otherwise.

In version 2.2.2 and later, Cumulus Linux uses the default Debian Linux arp_filter setting of 0. Versions of Cumulus Linux earlier than 2.2.0 also set the arp_filter parameter to 0, but version 2.2.0 and version 2.2.1 had the arp_filter parameter set to 1, which caused some inadvertent misbehavior in some circumstances.

The arp_filter is primarily used when multiple interfaces reside in the same subnet and is used to allow/disallow which interfaces respond to ARP requests. In the case of OSPF using IP unnumbered interfaces, many interfaces appear to be in the same subnet (actually containing the same address). If multiple interfaces are used between a pair of routers, having arp_filter set to 1 causes forwarding to fail.

Starting with Cumulus Linux 2.2.2, the arp_filter parameter is set to allow a response on any interface in the subnet and we use the arp_ignore setting (below) to limit cross-interface ARP behavior.

arp_ignore - INTEGER

Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:

0 - (default) Reply for any local target IP address, configured on any interface.

1 - Reply only if the target IP address is local address configured on the incoming interface.

2 - Reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface.

3 - Do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied.

4-7 - Reserved

8 - Do not reply for all local addresses.

The max value from conf/{all,interface}/arp_ignore is used when ARP request is received on the {interface}

The default Debian arp_ignore parameter allows the device to reply to an ARP request for any IP address on any interface. While this matches the expectation that an IP address belongs to the device, not an interface, it can cause some unexpected and undesirable behavior on a router.

For example, with the arp_ignore parameter set to 0, if an ARP request is received on one interface for the IP address residing on a different interface, the switch will respond with an ARP reply even if the interface of the target address is down. This can cause a loss of traffic due to incorrect understanding about the reachability of next-hops, and also makes troubleshooting extremely challenging in some failure conditions.

Cumulus Linux has set the arp_ignore value to 1 so that it will only reply to ARP requests on the interface which contains the target IP address. This acts much more like a traditional router and provides simplicity in troubleshooting and operation.

arp_notify - BOOLEAN

Define mode for notification of address and device changes.

0 - (default) Do nothing.

1 - Generate gratuitous arp requests when device is brought up or hardware address changes.

The default Debian arp_notify setting is to remain silent when an interface is brought up or the hardware address is changed. Since Cumulus Linux often acts as a next-hop for many end hosts, it immediately notifies attached devices when an interface comes up or the address changes. This speeds up convergence on the new information and provides the most rapid support for changes.

Where Tunable ARP Parameter Changes Have Been Implemented in Cumulus Linux

There are several places that can be used to implement the ARP parameter changes, including:

/proc/sys/net/ipv4/conf/all/arp*  (all interfaces)

/proc/sys/net/ipv4/conf/default/arp*  (default for future interfaces)

/proc/sys/net/ipv4/conf/swp*/arp*  (individual interfaces)

The ARP parameter changes in Cumulus Linux 2.2.2 are using the default file locations.

The all and default locations sound similar, with the exception of which interfaces are impacted, but they have significantly different operation. The all location can potentially change the value for all interfaces running IP, both now and in the future. The reason for the potentially is that the all value is applied to each parameter using either MAX or OR logic between the all and any port-specific settings, as the following table shows:

ARP Parameter Condition
arp_accept OR
arp_announce MAX
arp_filter OR
arp_ignore MAX
arp_notify MAX

For example, if the /proc/sys/net/conf/all/arp_ignore value is set to 1 and the /proc/sys/net/conf/swp1/arp_ignore value is set to 0 to try to disable it on a per-port basis, interface swp1 will still use the value of 1 in its operation. While it may appear that the port-specific setting should override the global all setting, that is not actually the way it works. Instead, the MAX value between the all value and port-specific value defines the actual behavior. This lack of simplicity has led us to implement the ARP parameter changes using the default location, instead.

The default location, /proc/sys/net/ipv4/conf/default/arp*, defines the values for all future IP interfaces. Changing the default setting of an ARP parameter does not impact interfaces that already contain an IP address. If changes are being made to a running system that already has IP addresses assigned to it, port-specific settings should be used instead.

Note: The way the default setting is implemented in Linux, the value of the default parameter is copied to every port-specific location, excluding those that already have an IP address assigned, as previously mentioned. Therefore, there is not any complicated logic between the default setting and the port-specific setting like there is when using the all location. This makes the application of particular port-specific policies much simpler and more deterministic.

To determine the current ARP parameter settings for each of the the locations, use the following mechanism (other methods are available, but this one is quite simple):

cumulus@switch:/$ sudo grep . /proc/sys/net/ipv4/conf/all/arp*

cumulus@switch:/$ sudo grep . /proc/sys/net/ipv4/conf/default/arp*

cumulus@switch:/$ sudo grep . /proc/sys/net/ipv4/conf/swp1/arp*

Note that Cumulus Linux implements this change at boot time using the file arp.conf at the following location:

cumulus@switch:/$ cat /etc/sysctl.d/arp.conf
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.default.arp_notify = 1

Changing Port-specific ARP Parameters

To change one of the port-specific ARP parameters in a running device, the simplest way to do it is with the following command:

cumulus@switch:/$ sudo sh -c "echo 0 > /proc/sys/net/ipv4/conf/swp1/arp_ignore"
cumulus@switch:/$ sudo grep . /proc/sys/net/ipv4/conf/swp1/arp*

To make the change persist through re-boots, edit the /etc/sysctl.d/arp.conf file and add your port-specific ARP setting.
Have more questions? Submit a request


This support portal has moved

Cumulus Networks is now part of the NVIDIA Networking Business Unit! The NVIDIA Cumulus Global Support Services (GSS) team has merged its operations with the NVIDIA Mellanox support services team.

You can access NVIDIA Cumulus support content from the Mellanox support portal.

You open and update new cases on the Mellanox support portal. Any previous cases that have been closed have been migrated to the Mellanox support portal.

Cases that are still open on the Cumulus portal will continue to be managed on the Cumulus portal. Once these cases close, they will be moved to the Mellanox support portal.

Powered by Zendesk