ovs internal port / veth pairs bandwidth limitation

Question

I have been doing some experiments on ovs these days. I have 2 physical machines with openstack running on it, and GRE tunnel is configured. I add 2 internal ports on br-int (integration bridge) of each machine and assign them to different namespace(ns1, ns2, ns3, ns4) and ip from same subnet(172.16.0.200,172.16.0.201,172.16.0.202,172.16.0.203). After configuration is done, VM(in same subnet)<-> virtual ports , virtual port <->virtual port on same/different nodes are all reachable(Use ping to test). However, weird thing shows up: I have used iperf to test the bandwidth, testing result shows as following:

1. Physical node<-> Physical node: 1GB/s
2. VM<->VM on same machine: 10GB/s
3. VM<->VM on different machines: 1GB/s
4. VM<->Virtual port same machine: 10GB/s
5. VM<->Virtual port different machines: 1GB/s
6. Virtual port<->Virtual port same machine: 16GB/s
7. Virtual port<->Virtual port different machines: 100~200kb/s (WEIRD!)

I have tried replace internal port with veth pairs, same behavior shows up. As I expect, the veth pair should behave similar to a VM because they both have separate namespace , and openstack VM uses same way (Veth pairs) to connect to br-int. But the experiment shows that the VM(node1) -> Virtual port(node2) has 1GB/s bandwidth but Virtual port(node1) -> Virtual port(node2) only has 100kb/s ? Anybody has any idea?

Thanks for your help.

Answer 1

When using GRE (or VXLAN, or other overlay network), you need to make sure that the MTU inside your virtual machines is smaller than the MTU of your physical interfaces. The GRE/VXLAN/etc header adds bytes to outgoing packets, which means that an MTU sized packet coming from a virtual machine will end up larger than the MTU of your host interfaces, causing fragmentation and poor performance.

This is documented, for example, here:

Tunneling protocols such as GRE include additional packet headers that increase overhead and decrease space available for the payload or user data. Without knowledge of the virtual network infrastructure, instances attempt to send packets using the default Ethernet maximum transmission unit (MTU) of 1500 bytes. Internet protocol (IP) networks contain the path MTU discovery (PMTUD) mechanism to detect end-to-end MTU and adjust packet size accordingly. However, some operating systems and networks block or otherwise lack support for PMTUD causing performance degradation or connectivity failure.

Ideally, you can prevent these problems by enabling jumbo frames on the physical network that contains your tenant virtual networks. Jumbo frames support MTUs up to approximately 9000 bytes which negates the impact of GRE overhead on virtual networks. However, many network devices lack support for jumbo frames and OpenStack administrators often lack control over network infrastructure. Given the latter complications, you can also prevent MTU problems by reducing the instance MTU to account for GRE overhead. Determining the proper MTU value often takes experimentation, but 1454 bytes works in most environments. You can configure the DHCP server that assigns IP addresses to your instances to also adjust the MTU.