I am using iperf3 verson 3.0.11 to measure the throughput of a UDP stream between two virtual machines. The virtual machines are deployed via OpenStack. Of the physical hosts, one is located in Bratislava, the other one in Ljubljana.
Network wise, the physical hosts each have a 1Gbit NIC and the VMs are connected via pci_passthrough, giving them full access to the NIC.
The iperf server displays the following output:
root@lju:/home/gts# iperf3 -s -V
iperf 3.0.11
Linux lju 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Fri, 15 Jan 2016 12:16:20 GMT
Accepted connection from 172.16.0.78, port 40774
Cookie: bra.1452860180.557437.6c142f196ae2a4
[ 5] local 172.16.0.80 port 5201 connected to 172.16.0.78 port 57847
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
iperf3: OUT OF ORDER - incoming packet = 70 and received packet = 97 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 71 and received packet = 97 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 72 and received packet = 99 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 14 and received packet = 123 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 15 and received packet = 125 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 78 and received packet = 137 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 79 and received packet = 137 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 80 and received packet = 139 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 82 and received packet = 172 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 83 and received packet = 173 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 65 and received packet = 175 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 66 and received packet = 176 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 67 and received packet = 177 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 84 and received packet = 177 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 86 and received packet = 198 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 18 and received packet = 203 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 19 and received packet = 205 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 22 and received packet = 270 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 23 and received packet = 273 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 38 and received packet = 349 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 39 and received packet = 350 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 40 and received packet = 350 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 41 and received packet = 350 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 47 and received packet = 412 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 48 and received packet = 412 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 49 and received packet = 412 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 50 and received packet = 414 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 55 and received packet = 490 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 56 and received packet = 490 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 57 and received packet = 490 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 60 and received packet = 541 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 61 and received packet = 541 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 62 and received packet = 541 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 63 and received packet = 541 AND SP = 5
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 105 MBytes 880 Mbits/sec 0.086 ms 141304/154697 (91%)
[ 5] 1.00-2.00 sec 114 MBytes 956 Mbits/sec 0.142 ms 164060/178642 (92%)
[ 5] 2.00-3.00 sec 114 MBytes 956 Mbits/sec 0.083 ms 165925/180507 (92%)
[ 5] 3.00-4.00 sec 114 MBytes 956 Mbits/sec 0.090 ms 164512/179094 (92%)
[ 5] 4.00-5.00 sec 114 MBytes 956 Mbits/sec 0.107 ms 166389/180972 (92%)
[ 5] 5.00-6.00 sec 114 MBytes 956 Mbits/sec 0.079 ms 159459/174041 (92%)
[ 5] 6.00-7.00 sec 114 MBytes 956 Mbits/sec 0.084 ms 168138/182720 (92%)
[ 5] 7.00-8.00 sec 114 MBytes 956 Mbits/sec 0.098 ms 169098/183680 (92%)
[ 5] 8.00-9.00 sec 114 MBytes 956 Mbits/sec 0.101 ms 166463/181046 (92%)
[ 5] 9.00-10.00 sec 114 MBytes 956 Mbits/sec 0.097 ms 168547/183128 (92%)
[ 5] 10.00-10.15 sec 16.6 MBytes 952 Mbits/sec 0.076 ms 23353/25473 (92%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-10.15 sec 13.8 GBytes 11.7 Gbits/sec 0.076 ms 1657248/1804000 (92%)
[SUM] 0.0-10.1 sec 1657248 datagrams received out-of-order
CPU Utilization: local/receiver 11.9% (1.3%u/10.6%s), remote/sender 44.5% (4.0%u/40.5%s)
iperf 3.0.11
Linux lju 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
The client displays this output:
root@bra:/home/gts# iperf3 -u -c 172.16.0.80 -b 0 -V
iperf 3.0.11
Linux bra 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Time: Fri, 15 Jan 2016 12:16:20 GMT
Connecting to host 172.16.0.80, port 5201
Cookie: bra.1452860180.557437.6c142f196ae2a4
[ 4] local 172.16.0.78 port 57847 connected to 172.16.0.80 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.38 GBytes 11.8 Gbits/sec 180260
[ 4] 1.00-2.00 sec 1.37 GBytes 11.7 Gbits/sec 179180
[ 4] 2.00-3.00 sec 1.37 GBytes 11.8 Gbits/sec 179950
[ 4] 3.00-4.00 sec 1.37 GBytes 11.8 Gbits/sec 180030
[ 4] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec 179240
[ 4] 5.00-6.00 sec 1.33 GBytes 11.5 Gbits/sec 174840
[ 4] 6.00-7.00 sec 1.40 GBytes 12.0 Gbits/sec 183040
[ 4] 7.00-8.00 sec 1.40 GBytes 12.1 Gbits/sec 183970
[ 4] 8.00-9.00 sec 1.38 GBytes 11.9 Gbits/sec 180830
[ 4] 9.00-10.00 sec 1.39 GBytes 12.0 Gbits/sec 182660
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 13.8 GBytes 11.8 Gbits/sec 0.076 ms 1657248/1804000 (92%)
[ 4] Sent 1804000 datagrams
CPU Utilization: local/sender 44.5% (4.0%u/40.5%s), remote/receiver 11.9% (1.3%u/10.6%s)
The first thing that is out of the ordinary is the amount of OUT OF ORDER packets incoming to the server when starting the measurement. Furthermore, the summary displays a bandwith of 11.7Gbit/s while the NIC is only capable of 1Gbit/s. The intermediate results show more potential on the server side, however on the client side the same abnormal number is shown. Consquently there also seems to be a loss of 92% of the datagrams.
Does anyone have pointers as to why iperf3 is exhibiting this behaviour?
