0

I have a KVM host with three NICs and can't get traffic between guests (in same subnet) above 1gbps aggregate. Is this by design and the only way to achieve >1gbps to use multiqueue support?

So far, I've tried all combinations of bonding/macvtap/VLANs (bond0->bond0.11/bond0.12->macvtap1/macvtap2->(Two hosts in VLAN11)/(Two Hosts in VLAN12). Macvtap in bridged mode.

I then tried creating VLANs on plain eth0-2 interfaces, adding bridges to those VLAN devices, but no throughput difference.

What I'm doing to test:

Host 1 & 2 in network 10.10.10.0/24

host1 > dd if=/dev/zero | ssh host2 "dd of=/dev/null"

By itself, the above gets a full 1gb

Then, on host 3 & 4 in network 10.10.11.0/24

host 3 > dd if=/dev/zero | ssh host4 "dd of=/dev/null"

Throughput drops on each down to 500mbps for a total of 1gbps.

The only thing I haven't tried is macvtap in passthrough mode to force the switch to handle it.

Any thoughts on achieving the full 3gbps of links I have available? Better yet, any way to exceed the capacity of my aggregate links with "fake" interfaces?

RyanH
  • 367
  • 1
  • 6
  • if you're running only one `dd` command, don't expect the aggregate to utilize more than one NIC. Run multiple io generating threads to see better results – dyasny Feb 18 '15 at 00:42
  • I think you're right. I reconfigured my 3 NIC bond and let libvirt configure the macvtaps. Simultaneous transfers via dd over ssh began giving me 70MB/s per guest pair (x3) so 210MB/s total. I then installed iperf which was showing 10Gbits/sec per guest pair. Now I'm checking netcat which when running on one guest pair, they're getting 450MB/s. Better. – RyanH Feb 18 '15 at 00:58
  • Of course I'm right. Moreover, no matter what bonding mode you use (and you failed to mention such an insignificant detail), this will still be the case. BTW, avoid modes 0 and 6 with bridges, they will not work. – dyasny Feb 18 '15 at 02:07

1 Answers1

2

Your SSH transfer will be CPU-bound, limited by single-threaded encryption. Use iperf or netperf or something else designed for testing just network bandwidth.

suprjami
  • 3,536
  • 21
  • 29