Throughput Testing and Troubleshooting

Follow

This guide is intended to help you in cases where you believe that the throughput on your Cumulus Linux switch is underperforming.

{{table_of_contents}}

Traffic Profile

First you need to understand the traffic profile. You need to gather the following information:

  • What is the source/destination of the connection?
    • Bare metal or VM?
    • OS
    • IP address
    • MAC address
    • CPU information
    • RAM
  • What is the full traffic path of the connection?
    • L1 connections
    • L2 hops
    • L3 hops
  • What service, protocol and ports are being used for the connection?
    • TCP/UDP
    • Non-standard ports
    • Common throughput testing - HTTP, FTP, high port TCP
  • How many concurrent connections have been established between the client and server?
  • What is the average packet size?
  • TCP specific questions:
    • What is the window?
    • Is there any packet reordering?
    • Is there packet loss and what is the recovery mechanism?
    • Are we using SACK or normal ACK?
  • How much other traffic is going through the Cumulus Linux switch?
    • What is the speed of the port?

Network Testing with Open Source Tools

Testing network performance requires generating traffic at various rates and measuring throughput, loss, latency, jitter, and so forth. This article covers multiple open source network test tools, which are available for free. Running the software is only half of the story however. Tuning the host to efficiently generate and receive network traffic as well as monitoring the hardware load is equally important. All of the tools explored here have different features, statistics reporting and operation.

Generating traffic at data rates above 1Gbps requires fairly powerful server hardware. Without going into the depths of computer architecture, the CPU, memory, PCI bus and network interface controller (NIC) all play a part in generating and receiving network data. Typically the bottleneck is the CPU core executing the client or server processes. The PCI bus may also constrain the performance depending on the PCI version. Finally, the NIC must be able to support the traffic attempting to be sent.

Many of the examples and tuning information were learned from Energy Sciences Network. They go into a lot more detail about the host tuning and the various capabilities of the different tools.

iPerf3

This is a complete rewrite of the original iPerf tool. It has many of the same features, although it does not yet do multicast. Below are some basic commands to get started, which are the same options available in version 2. For bandwidth testing, iPerf3 is preferred over iPerf1 or 2.

Note: Cumulus Networks tested iPerf3 and identified some functionality issues on Debian Wheezy 7.9, while we noticed no issues on Debian Jessie 8.3. iPerf versions 1 and 2 work on Debian Wheezy.

Server Commands

Start the server on the default port for IPv4 (default):

iperf3 -s 

Start the server in daemon mode:

iperf3 -s -D

Start the server on port 5003:

iperf3 -s -p 5003

Start using 0.5 second interval times:

iperf3 -s -i 0.5

Client Commands

Run a 30 second test, giving results every 1 second:

iperf3 -c <dst-ip> -i 1 -t 30 

Note: Using granular interval reporting counters can help provide greater information when examining why a specific throughput test is slow.

Run a test from remotehost to localhost:

iperf3 -c <dst-ip> -i 1 -t 20 -R 

Run a test with 4 parallel streams, and with a 32M TCP buffer

iperf3 -c <dst-ip> -w 32M -P 4 

Note: Window size is very important for TCP tests. The TCP window dictates how many bytes and packets are allowed in flight before receiving an ACK. A small window may result in high speeds for a directly connected server and client, but low speeds for a server and client across a network with high latency between the client and server. As RTT increases, the time between packet and ACK increases, which results in more packets in flight. If the TCP window is too small to accommodate, the sender will scale back the speed at which the packets are sent in order to not exhaust the window. This becomes even more important when using jumbo frames. 

Note: Utilizing parallel sessions can help improve throughput numbers when using load balancing features such as LACP bonded links, MLAG switches or ECMP routing. In these traffic load balancing features, flows are normally hashed using the source and destination IP address and port. This means that a single flow will always be hashed to the same physical link and path. In order to test the full utilization of the traffic path, the testing methodology should utilize multiple IP addresses and ports for the client and server connections. The throughput results for each concurrent connection should be added together to get the total bandwidth of the infrastructure. The -P option creates multiple sessions with varying source ports which, depending on the traffic hashing algorithm, can result in greater throughput results.

Run a 200 Mbps UDP test:

iperf3 -c <dst-ip> -u -i 1 -b 200M

Note: When testing with UDP, the -b modifier can be used to set a target throughput rate. Use -b 0 for an unlimited bandwidth target. For UDP, without setting this value the throughput values may be very low.

Note: Many client/server NIC cards perform TCP checksum/header offloading. This means that these calculations are done on the NIC itself. But for UDP, header processing is performed in software. As a result, performance numbers between UDP and TCP may not be consistent.

Optimizations

iPerf3 features many optimizations that work well with modern server hardware. It also has new features to improve how statistics are measured.

Run the test for 2 seconds before collecting results, to allow for TCP slowstart to finish. The -O flag sets the Omit mode to a 2 second delay, and the -i flag sets a ½ second report interval.

iperf3 -c <dst-ip> -O 2 -i 0.5

Using the -Z flag enables the Zero Copy mode, which uses the sendfile() system call. This uses much less CPU to put the data on the PCI bus.

iperf3 -c <dst-ip> -Z

Output the results in JSON format using the -J flag for easy parsing by most coding languages. This output is printed after the entire test finishes running.

iperf3 -c <dst-ip> -J 

Set the CPU affinity for the sender (-A 2) or the sender, receiver (-A 2,3), where the cores are numbered starting at 0. This has the same effect as running numactl -C 4 iperf3.

iperf3 -c <dst-ip> -A 2,3 

Note: It is not necessary to use -A x,y if you are using the -P flag. The -P flag creates multiple connections and the server side appears to balance the threads across all available cores.

Caveats

As identified in the notes above, iPerf as a tool, in general, can give a vastly varied set of results depending on the settings. However, iPerf has some issues that make it difficult to use for performance testing on modern networks. Factors that matter in testing:

  • TCP window size, controlled using the -w flag
  • TCP vs UDP procotol, controlled using the -u modifier
  • Target throughput rate, controlled using the -b modifier
  • CPU utilization, controlled using the -Z modifier and the -A modifier
  • Data time sets used for average calculations, controlled using -O modifier
  • Concurrent connections on port, controlled using the -P modifier

By default iPerf uses TCP/UDP port 5201/5001 (depending on version) for ports during transfer. This does not work too well for ECMP testing. Specifying the -P flag tells iPerf to spawn multiple client threads, where the source port uses 5201 as well as a series of ephemeral ports. But when using this method, it is also normal to see a lot of these messages: bind failed: Address already in use

iPerf (version 2)

When installing iPerf from the distribution's repository, it is most likely version 2. This is a rather old piece of software, and is not really maintained anymore.

Multicast

One unique feature of iPerf2 to take a look at is the ability to send multicast traffic. Here are the client and server commands to use:

iperf -c 224.0.0.1 -u -b 512k  #source
iperf -B 224.0.0.1 -su         #receiver

This causes the server to listen on a group address, meaning it sends an IGMP report to the connected network device. On the client side, be sure to create a static route for the multicast range, with a next-hop toward the network devices under test (let's assume this is out from eth1).

ip route add 224.0.0.0/4 dev eth1

Caveats

The results are inconsistent for UDP above 2Gbps. We recommend using nuttcp or iPerf3 for high speed UDP testing. However, it is perfectly fine for testing networks if the traffic rate is kept below 2Gbps. Above that, the statistics are completely unreliable. Therefore, we are not going to cover this as a viable performance test tool.

nuttcp

This tool evolved over the past 30 years, and has several useful features. It supports rate limiting, multiple parallel streams and timer based execution. It also includes IPv6, IPv4 multicast and the ability to set parameters, such as TCP MSS or TOS/DSCP bits.

Looking at the application's help file, it's easy to see that this tool has a lot of unique functionality. One thing to understand is there are client (transmitter/receiver) and server entities. The server settings can be controlled from the client command, for such things as TCP window size. Server processes automatically fork into the background and run until an error condition or are killed. The examples below demonstrate some of the basics.

Server Commands

Start the server on the default port (5000):

nuttcp -S

Start server on the default port for IPv6:

nuttcp -s -6

Start server on port 5003:

nuttcp -S -P 5003

Transmitter Commands

Run a 10 second test, giving results every second:

nuttcp -i 1 <server>

Running longer tests require the -T flag, where you can specify duration like this:

-T30 = 30 sec, -T1m = 1 min, and -T1h = 1 hr

Run a 10 second test, setting the transmitter/receiver window as well as the server receive/transmit window size to 32KB. You can also set the window in Megabytes (-w5m) or Gigabytes (-w5g):

nuttcp -w32 -ws32 <server>

The default traffic uses TCP, but it can also be run as UDP with 8192 byte datagrams (default):

nuttcp -u <server>

Run a 10 second UDP test at 50Mbps, and report throughput and packet loss:

nuttcp -u -R50m <server>

Run a 10 second UDP test with 1200 byte datagrams:

nuttcp -u -l1200 <server>

Receiver Commands

Run a 10 second test in reverse (server → client), giving results every 1 second:

nuttcp -r -i 1 <server>

Special Features

UDP Burst Mode

Starting with version 6.2.8, nuttcp includes a burst mode for UDP, which is useful for finding network paths constrained a lack of buffering. To do this, set an instantaneous data rate with optional packet burst. Specify the rates in Kbps (-R100), Mbps (-R5m), Gbps (-R5g) or PPS (-R1000p). The burst is just a count of packets.

Send 300 Mbps of UDP at an instantaneous rate in bursts of 20 packets for 5 seconds:

nuttcp -u -Ri300m/20 -i 1 -T5 <server>

This test may show a small amount of loss. If you increase the burst size at the same instantaneous rate, it will increase the stress to the network buffering. This will progressively show more loss as the transit device overflows its buffers.

On 10GbE network with 9000 byte MTU, we can send 8972 byte UDP datagrams at an instantaneous rate of 300 Mbps, and bursting at 100 packets. This should be lossless:

nuttcp -l8972 -T30 -u -w4m -Ri300m/100 -i1 <server>

Again, increasing the burst value and retesting eventually shows where buffer exhaustion happens in the network:

10G UDP Testing

It definitely seems like nuttcp the best tool for high speed UDP testing. It took very little effort to create a full 10 Gbps traffic flow using UDP, but it requires jumbo MTU size packets (9K):

nuttcp -l8972 -T30 -u -w4m -Ru -i1 <server>

CPU Affinity

One of the newer features introduced in version 7.1 is CPU affinity. This allows the user to specify on which core to execute the process.

Sending multiple streams, you can use CPU affinity binding to make sure both processes do not end up on the same CPU core. Be sure to start the servers on the intended destination ports. Also, using the statistics output identifier helps to label the results with an s1 or s2:

nuttcp -i1 -xc 2/2 -Is1 -u -Ru -l8972 -w4m -p 5500 <server> & \
nuttcp -i1 -xc 3/3 -Is2 -u -Ru -l8972 -w4m -p 5501 <server>

Bandwidth Controller (bwctl)

This tool is a wrapper for iPerf2, iPerf3 and nuttcp. The documentation for this wrapper is available here.

The advantage of using bwctl is that it is a centralized bandwidth testing tool that utilizes all the tools listed above to provide bandwidth test results. You can read the installation guide as a PDF or on the Web.

Ostinato

Ostinato is an open source tool for generating packets. It operates primarily from a GUI, but is easily installed in a Linux desktop environment.

Although there is no CLI, it does have a Python API that you can use to generate packets.

Cumulus Linux Switch Outputs

When performing bandwidth testing, it is valuable to cross reference the results from the testing tool against various outputs on the Cumulus Linux switch.

Traffic Services on the Switch

Throughput levels may change based on what services are applied to the traffic. Ideally, all traffic should be hardware switched through the Broadcom ASIC. But certain services applied to the traffic require the traffic to be punted from the ASIC to the switch CPU. Since the switch CPU is slower at processing and forwarding packets, the throughput may suffer from these features. Some features that may cause software processing of packets are:

  • Encapsulation methodologies
  • GRE
  • NAT
  • PBR

It is also important to look at what general features are applied to the traffic:

  • Layer 2 vs. layer 3 switching
  • Netfilter ACLs for traffic

Troubleshooting Output

You should examine these two outputs to verify the throughput:

  • cl-netstat
  • ethtool -S swpX

cl-netstat

This command is a wrapper for the interface counters provided in Linux. The output of ip -s link show up prints counters of all TX and RX packets. But these native Linux commands have no way to clear counters. The cl-netstat command adds this functionality, and can track individual connections under controlled environments. The example below shows the results of an iPerf3 test run between a client and server through the Cumulus Linux switch:

Client iPerf3 Results

root@CLIENT:~# iperf3 -c 10.1.1.100
Connecting to host 10.1.1.100, port 5201
[  4] local 10.1.2.100 port 53984 connected to 10.1.1.100 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   980 MBytes  8.22 Gbits/sec    0   5.17 MBytes       
...   
[  4]   9.00-10.00  sec   951 MBytes  7.98 Gbits/sec    0   5.17 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.32 GBytes  8.01 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  9.29 GBytes  7.98 Gbits/sec                  receiver

cl-netstat Results

cumulus@switch:~$ sudo cl-netstat   
Kernel Interface table
Iface      MTU    Met    RX_OK    RX_ERR    RX_DRP    RX_OVR    TX_OK    TX_ERR    TX_DRP    TX_OVR  Flg
-------  -----  -----  -------  --------  --------  --------  -------  --------  --------  --------  -----
swp13     1500      0    62622         0         0         0  6904270         0         0         0  BMRU
swp14     1500      0  6904271         0         0         0    62623         0         0         0  BMRU

In the above example, we can see that traffic has successfully flowed between swp13 and swp14 without dropping any packets.

ethtool -S

Similarly, the output of ethtool -S can be used to cross reference the output obtained from cl-netstat.

cumulus@switch:~$ sudo ethtool -S swp13
NIC statistics:
     HwIfInOctets: 5583156718
     HwIfInUcastPkts: 5140969
     HwIfInBcastPkts: 14
     HwIfInMcastPkts: 9272
     HwIfOutOctets: 210229661164
     HwIfOutUcastPkts: 139033950
     HwIfOutMcastPkts: 17657
     HwIfOutBcastPkts: 8556
     HwIfInDiscards: 0
     HwIfInL3Drops: 0
     HwIfInBufferDrops: 0
     HwIfInAclDrops: 63
     HwIfInDot3LengthErrors: 0
     HwIfInErrors: 0
     SoftInErrors: 0
     SoftInDrops: 0
     SoftInFrameErrors: 0
HwIfOutDiscards: 0
HwIfOutErrors: 0
HwIfOutQDrops: 0
HwIfOutNonQDrops: 0
cumulus@switch:~$ sudo ethtool -S swp14
NIC statistics:
     HwIfInOctets: 210229250162
     HwIfInUcastPkts: 139051524 
     HwIfInBcastPkts: 5 
     HwIfInMcastPkts: 9230 
     HwIfOutOctets: 5583641007 
     HwIfOutUcastPkts: 5124014 
     HwIfOutMcastPkts: 17875 
     HwIfOutBcastPkts: 8711 
     HwIfInDiscards: 0 
     HwIfInL3Drops: 0 
     HwIfInBufferDrops: 0 
     HwIfInAclDrops: 54 
     HwIfInDot3LengthErrors: 0 
     HwIfInErrors: 0 
     SoftInErrors: 0 
     SoftInDrops: 0 
     SoftInFrameErrors: 0
HwIfOutDiscards: 0
HwIfOutErrors: 0
HwIfOutQDrops: 0
HwIfOutNonQDrops: 0 

The counters in ethtool are much larger than the output of cl-netstat because the ethtool counters cannot be cleared. The key output to examine is to verify whether errors in receive or transmit are increasing.

The directionality of the counters will matter as well. Drops on ingress via the HwIfIn prefix indicate that traffic rate on receive is too fast. This generally indicates a bursty traffic or choke points further along the traffic path. Drops on egress via the HwIfOut prefix indicate a many-to-one problem, where multiple RX interfaces are trying to send traffic out a single TX interface. This is not always a bad result, as it indicates full bandwidth utilization by the interface.

RX Errors

If the counters under RX_ERR are increasing during the bandwidth test, there may be problems with packet forwarding through the switch. Here are some previously written articles that address this scenario:

 

Have more questions? Submit a request

Comments

Powered by Zendesk