Setup private networking between two hosts and two VMs with libvirt / openvswitch

Question

I have two nodes and two VM's

kvm01
-nic1 123.123.123.1

VM1 
-vnet1 123.123.123.2 (public) -vnet2 10.0.0.1 (private)

kvm02
-nic1 123.123.123.2

VM2
-vnet1 123.123.123.4 (public) -vnet2 10.0.0.2 (private)

How is it possible to setup an openvswitch network so that VM1 and VM2 are on a private network however the hosts are in two different locations?

EDIT:

Running this command:

# ovs-vsctl add-port br-private vxlan1 -- \
  set Interface vxlan1 type=vxlan options:remote_ip=123.123.123.2

Resulted in this OVS configuration:

root@backup01:~# ovs-vsctl show
6276bd0a-920b-469d-a4e0-90d990dd8f94
    Bridge "br-private1"
        Port "vxlan1"
            Interface "vxlan1"
                type: vxlan
                options: {remote_host="107.150.29.72"}
        Port "br-private1"
            Interface "br-private1"
                type: internal
    ovs_version: "2.3.0"
root@kvmssd01:~# ovs-vsctl show
da6399d4-1435-437d-90d7-3e75c443389b
    Bridge br-private
        Port "vxlan1"
            Interface "vxlan1"
                type: vxlan
                options: {remote_host="107.150.29.68"}
        Port br-private
            Interface br-private
                type: internal
    ovs_version: "2.3.0"

Using this kernel:

root@backup01:~# uname -r
3.16.0-4-amd64

With this module:

root@backup01:~# modinfo openvswitch
filename:       /lib/modules/3.16.0-4-amd64/kernel/net/openvswitch/openvswitch.ko
license:        GPL
description:    Open vSwitch switching datapath
depends:        libcrc32c,vxlan,gre
intree:         Y
vermagic:       3.16.0-4-amd64 SMP mod_unload modversions

And this version of OVS:

root@backup01:~# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.3.0
Compiled Dec 19 2014 03:59:10
DB Schema 7.6.0

Resulted in these errors:

root@backup01:~# ovs-vswitchd logs
2015-07-23T16:34:59Z|00001|reconnect|INFO|logs: connecting...
2015-07-23T16:34:59Z|00002|reconnect|INFO|logs: connection attempt failed
  (Address family not supported by protocol)
2015-07-23T16:34:59Z|00003|reconnect|INFO|logs: waiting 1 seconds before reconnect
2015-07-23T16:35:00Z|00004|reconnect|INFO|logs: connecting...
2015-07-23T16:35:00Z|00005|reconnect|INFO|logs: connection attempt failed 
    (Address family not supported by protocol)
2015-07-23T16:35:00Z|00006|reconnect|INFO|logs: waiting 2 seconds before reconnect

lsmod output:

# lsmod | grep openvswitch 
openvswitch 63932 0 
gre 12777 1 openvswitch 
vxlan 35053 1 openvswitch 
libcrc32c 12426 1 openvswitch

`root@kvmssd01:~# lsmod | grep openvswitch openvswitch 63932 0 gre 12777 1 openvswitch vxlan 35053 1 openvswitch libcrc32c 12426 1 openvswitch ` — Charlie, Jul 23 '15 at 17:51
hmmm, seems even *before* I setup the vlan it shows that error. default install on debian 8. maybe it's something wrong with openvswitch's server? — Charlie, Jul 23 '15 at 18:08

larsks · Accepted Answer · 2015-07-22T17:01:48.510

7

One option is to set up a VXLAN tunnel between OVS bridges on the two hosts.

Creating the OVS bridges

On each host, create an OVS bridge that will be used by the private network:

ovs-vsctl add-br br-private

When you create your libvirt VMs, attach vnet2 on each guest to the br-private bridge. Using virt-install this would look something like:

virt-install ... -w bridge=br-private,virtualport_type=openvswitch

If you are using some other mechanism to create your guests, the corresponding XML looks like:

<interface type='bridge'>
    < source bridge='br-private'/>
    < virtualport type='openvswitch'/>
< /interface>

Creating the VXLAN tunnels

Now create a VXLAN tunnel from kvm01 to kvm02. On kvm01:

ovs-vsctl add-port br-private vxlan1 -- \
  set Interface vxlan1 type=vxlan options:remote_ip=123.123.123.2

And on kvm02:

ovs-vsctl add-port br-private vxlan1 -- \
  set Interface vxlan1 type=vxlan options:remote_ip=123.123.123.1

With this tunnel in place and your guests connected to br-private, you have created a virtual network that spans multiple hosts.

Note that VXLAN runs over UDP port 4789, so you may need to modify your firewall configuration to permit these connections.

The finished OVS configuration will look something like this:

# ovs-vsctl show
ac885d3d-b636-4bb1-a75e-37f361af87e3
    Bridge br-private
        Port "vxlan1"
            Interface "vxlan1"
                type: vxlan
                options: {remote_ip="192.168.122.107"}
        Port br-private
            Interface br-private
                type: internal
        Port vnet2
            Interface vnet2
    ovs_version: "2.3.2"

...although of course using your host ip addresses, rather than this address from my testing environment.

edited Jul 22 '15 at 17:01

answered Jul 22 '15 at 16:25

larsks

277,717
41
399
399

AWESOME! Thanks for the help. Still trying to grasp openvswitch. One last question. How many VLAN's could I potentionally have (for example, if I have 1000 VM's on one host all with private networks to 1000VM's on another host., would it be best / viable to have 1000 VLAN's?). – Charlie Jul 23 '15 at 15:31
Also, how could I setup a DHCP server on openvswitch (or maybe through libvirt?) so that hosts on libvirt / KVM are automatically assigned private IPs. THANKS FOR ALL THE HELP!! – Charlie Jul 23 '15 at 15:39
You can tunnel multiple VLANs over a single VXLAN tunnel, but that involves adding openflow rules that translate VLAN tags into VXLAN tunnel ids (and back on the other side). Both of these questions are somewhat beyond the scope of the original question, so I would suggest doing a little research and then maybe posting new questions (possibly on Serverfault, which is arguably more appropriate for the topic). – larsks Jul 23 '15 at 15:43
Note that OpenStack does both of these things (providing DHCP servers for private virtual networks and tunneling VLANs over VXLAN tunnels), so you may find some examples in OpenStack docs/blogs/etc. – larsks Jul 23 '15 at 15:44
I'm trying to set this up but it seems that I keep getting the error "Address family not supported by protocol" when I enter the port information. Any ideas? – Charlie Jul 23 '15 at 16:24
Please update your question with (a) the exact command line you are using, (b) the exact error message, and (c) information about what distribution, kernel, and version of openvswitch you are using. – larsks Jul 23 '15 at 16:26

Setup private networking between two hosts and two VMs with libvirt / openvswitch

1 Answers1

Creating the OVS bridges

Creating the VXLAN tunnels