Creating more than 20 VXLAN interfaces on linux

Question

How can I create more than 20 vxlan interfaces in Ubuntu/Debian?

In Linux kernel 3.16 I found that I can create more than 20 vxlan interfaces but they will not work properly as in sending arp messages does not work.
in Linux Kernel 4.4 I get this error message "RTNETLINK answers: No buffer space available" when creating the 21'st interface

I've tested this by creating a small shell script that creates the interfaces on a fresh ubuntu 14.04 & 16.04, also tested this on Debian 8.

The script testvxlan.sh looks like this:

#!/bin/bash

for i in {1..30}
do
    echo "Setting up interface br0.$i"
    ip link add br0.$i type vxlan id $i group 239.0.0.$i dev eth0 dstport 4789
    ip addr add 192.168.$i.1/24 dev br0.$i
    ip link set dev br0.$i up
    #ip link delete br0.$i
done

when running this on a fresh Ubuntu 16.04 it looks like this:

root@ubuntu-xenial:~# ./testvxlan.sh
Setting up interface br0.1
Setting up interface br0.2
Setting up interface br0.3
Setting up interface br0.4
Setting up interface br0.5
Setting up interface br0.6
Setting up interface br0.7
Setting up interface br0.8
Setting up interface br0.9
Setting up interface br0.10
Setting up interface br0.11
Setting up interface br0.12
Setting up interface br0.13
Setting up interface br0.14
Setting up interface br0.15
Setting up interface br0.16
Setting up interface br0.17
Setting up interface br0.18
Setting up interface br0.19
Setting up interface br0.20
Setting up interface br0.21
RTNETLINK answers: No buffer space available
Setting up interface br0.22
RTNETLINK answers: No buffer space available
Setting up interface br0.23
RTNETLINK answers: No buffer space available
Setting up interface br0.24
RTNETLINK answers: No buffer space available
Setting up interface br0.25
RTNETLINK answers: No buffer space available
Setting up interface br0.26
RTNETLINK answers: No buffer space available
Setting up interface br0.27
RTNETLINK answers: No buffer space available
Setting up interface br0.28
RTNETLINK answers: No buffer space available
Setting up interface br0.29
RTNETLINK answers: No buffer space available
Setting up interface br0.30
RTNETLINK answers: No buffer space available

How can I increase this bufferspace or is it even possible?

Torsten-Sven Urbas · Accepted Answer · 2017-01-19T16:49:24.277

Since you use multicast vxlan's, the limit is actually the maximum number of IGMP memberships:

[root@cpu1 ~]# cat /proc/sys/net/ipv4/igmp_max_memberships 
20

You can raise this limit and should be able to bring up more than 20 vxlans:

[root@cpu1 ~]# echo 100 >/proc/sys/net/ipv4/igmp_max_memberships

If you want this change to be persistent across reboots, you do need to add the following snippet to your /etc/sysctl.conf or /etc/sysctl.d/:

net.ipv4.igmp_max_memberships = 100

score 1 · Answer 2 · edited Feb 10 '17 at 12:33

We've been working extensively with multicast vxlans, having 200+ in a single node, with igmp_max_memberships set to 400...

Apart from a lot of igmp reports things tended to give igmp snooping problems on switches that can do that.

Therefore we created a new group every 256 new vxlans separating the tenants by the vni.

YMMV, but snooping is a good way to limit the multicast broadcast domain, even on elcheapo switches that have a limited snooping table. anyway the vni field is 16 bits, so you can easily fit 65535 vxlans in in one mc group.

for grp in `seq 1 4` ; do
   for vni in `seq 1 64` ; do
      echo ip link add vx-`printf "%04x" $(($grp*$vni))` type vxlan id $vni group 239.0.1.$grp dev bkpln dstport 4789
   done
done | sh -x

Creating more than 20 VXLAN interfaces on linux

2 Answers2