2

I'm experiencing extremely weak network performance when sending data from a host machine to a docker container, compared to traffic sent from a host machine to the same host.

I've measured the TCP performance using iPerf, these are the results on different operating systems with Docker installed:

Operating System Client Server Bandwidth (measured with iPerf, see below)
macOS Host Host ~56.3 Gbits/sec
macOS Host Container ~350 Mbits/sec
Windows Pro 10 Host Host ~5 Gbit/sec
Windows Pro 10 Host Container ~680 Mbit/sec

The command run to get the results:

iperf -s -p 8020 # on the server side (receiving data)
iperf -c -p 8020 # on the client side (sending data)

I'm running the Docker containers in networking bridge mode (default) and publishing the port 8020 (network host mode is not available for macOS).

These results seem very unintuitive: First of all, on both operating system the measured bandwidth between a Docker container and the host machine falls drastically below the available bandwidth of a host-to-host-connection. Why is that? Afaik packets addressed to localhost are generally not handled by the NIC but rather by the OS/kernel itself, so they shouldn't depend on the NIC's available bandwidth. Although each Docker container uses its own network interface, I don't think the network card is involved there either (which could be a possible cause for a limited bandwidth/poor performance).

Secondly, it seems weird that the average bandwidth on Windows is higher than on macOS for host-to-container connections, even though we've achieved a much higher bandwidth on macOS for host-to-host connections.

Why on earth do we experience such a poor bandwidth for host-to-container packets? Is this a known issue with Docker containers (we couldn't find anything related to this on the internet that has the same results cross-platform)?Is there a good way to circumvent/fix this issue?

j3141592653589793238
  • 1,810
  • 2
  • 16
  • 38

1 Answers1

0

The Maximum Transmission Unit (MTU) of the loopback interface tends to be rather large. For example, the "lo0" interface of my Mac running 12.4 is 16384 bytes. You might look at the MTUs of the Docker interfaces in comparison. TCP relies on large MTUs and/or stateless offloads (ChecKsum Offload/CKO and TCP Segmentation Offload/TSO and Large/Generic Receive Offload/[LG]RO) to achieve high bitrates by minimizing the number of trips up and down the protocol stack required to achieve a given bitrate.

If, for example, the MTU of the Docker interface(s) is 1500 bytes rather than 16384, and there aren't also stateless offloads, you would expect the Docker-based test to need roughly 16384/1500 or ~10 times as many trips up and down the protocol stack as a plain loopback. And, it will be a longer fundamental code path on top of that. The difference would be even greater if TSO/GRO were available via the loopback and not the Docket interfaces. With those in place a TCP can behave as if the MTU was up to 64 KiB.

If you run something like a netperf TCP_RR test in addition to your iperf bulk transfer test, you will get a first approximation of the difference in basic packet path length between the plain loopback and Docker configurations.

Rick Jones
  • 166
  • 4
  • I've just set the MTU of the Docker interface to 2^16 (and confirmed the updated MTU via ifconfig), however, for some reason the results are still the same - you got any idea why? Thanks a lot! – j3141592653589793238 Jul 06 '22 at 13:49
  • You could/should triple-check the TCP MSS followed the change in MTU - packet capture started just before your test so you can see the connection establishment packets. You might also snapshot netstat -s statistics before and after each type of test and look for differences in things like retransmission rates and whatnot. – Rick Jones Jul 06 '22 at 21:14