KVM bridge over bonding with multiple subnets

Question

I can access host via 213.xxx.xxx.1 assigned to br0:1 but VM with 213.xxx.xxx.2 (bridge interface on br0:1) doesn't work.

Do I have to create another bridge interface for the second subnet, but then bond0 is already connected with br0?!

Basically how I can use the second subnet for VMs in this scenario?

Subnets:

213.xxx.xxx.176/28
213.xxx.xxx.0/27 routed via 213.xxx.xx.180

Setup:

em[1-2]

# cat /etc/sysconfig/network-scripts/ifcfg-em[1-2]
DEVICE="em[1-2]"
NAME="bond0-slave[1-2]"
BOOTPROTO="none"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL=no
IPV6INIT=no
PEERDNS=no
MASTER=bond0
SLAVE=yes

bond0

# cat /etc/sysconfig/network-scripts/ifcfg-bond0 
DEVICE="bond0"
NAME="bond0"
BOOTPROTO="none"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Bond"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"
BONDING_OPTS="mode=active-backup miimon=100"
BRIDGE=br0

br0

# cat /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE="br0"
NAME="br0"
BOOTPROTO="none"
IPADDR="213.xxx.xxx.180"
GATEWAY="213.xxx.xxx.177"
NETMASK="255.255.255.240"
BROADCAST="213.xxx.xxx.191"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Bridge"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"

br0:1

# cat /etc/sysconfig/network-scripts/ifcfg-br0:1 
DEVICE="br0:1"
NAME="br0:1"
BOOTPROTO="none"
IPADDR="213.xxx.xxx.1"
NETMASK="255.255.255.224"
BROADCAST="213.xxx.xxx.31"
NM_CONTROLLED="yes"
#ONBOOT="yes"
TYPE="Bridge"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"
ONPARENT="yes"

Update Wed 18 Jan 16:25:17 GMT 2017:

<interface type='bridge'>
      <mac address='52:54:00:4c:4f:27'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Update Tue 24 Jan 11:44:21 GMT 2017:

VM network setup:

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
NAME="eth0"
DEVICE="eth0"
ONBOOT="yes"
IPADDR="213.xxx.xxx.2"
PREFIX="27"
GATEWAY="213.xxx.xxx.177" <-- not sure if this should be 213.xxx.xxx.180 instead?!

IPTables on host:

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68

# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
RETURN     all  --  192.168.122.0/24     224.0.0.0/24        
RETURN     all  --  192.168.122.0/24     255.255.255.255     
MASQUERADE  tcp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  udp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  all  --  192.168.122.0/24    !192.168.122.0/24

It works when I set a default gateway to 213.xxx.xxx.1 on VM (IP from br0:1 on the host) - not sure why I can't just use the same default gateway like host (213.xxx.xxx.177).

I had to also flush IPTables so I need to find out what rules are necessary.

I'm also getting redirection when pinging from VM so not sure if this config is optimal:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 213.xxx.xxx.1: icmp_seq=1 Redirect Host(New nexthop: 213.xxx.xxx.177)
From 213.xxx.xxx.1 icmp_seq=1 Redirect Host(New nexthop: 213.xxx.xxx.177)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=7.26 ms
...

Update Tue 24 Jan 14:49:43 GMT 2017:

I'm not sure why IPTables actually filter packets traversing the bridge even if br_netfilter module is not loaded

# lsmod | grep "br_netfilter"; echo $?
1
# test -d /proc/sys/net/bridge; echo $?
1

This rule also didn't help:

# iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

please show your vm configurations, your vm needs to use vnet insede the br0 — c4f4t0r, Jan 18 '17 at 16:01
the only thing that you need is configure the ip address inside the vm and set default gw, because you vm when start create inside br0 and br0 act as switch, forwarding the layer 2 traffic to your network — c4f4t0r, Jan 19 '17 at 00:56

score 1 · Answer 1 · edited Apr 13 '17 at 12:14

1

From Wikipedia:

A default gateway in computer networking is the node that is assumed to know how to forward packets on to other networks.

It's mean that the gateway node must be a node in network or subnet. You use two subnets on one interface br0 213.xxx.xxx.0/27 and 213.xxx.xxx.176/28.

There is gateway host 213.xxx.xxx.177 in network 213.xxx.xxx.176/28. And your host server gets internet access via this gateway. I think gateway host 213.xxx.xxx.177 is a router and this router isn't member (node) of subnet 213.xxx.xxx.0/27. But it knows that the subnet 213.xxx.xxx.0/27 is accessible via node 213.xxx.xxx.180 (your host server).

Your VM has wrong gateway address. If VM has ip address 213.xxx.xxx.2/27 then VM must have gateway address from same subnet 213.xxx.xxx.0/27. That is why VM must have gateway address 213.xxx.xxx.1, witch your host server has as secondary ip address on br0 interface.

If you will create VM in subnet 213.xxx.xxx.176/28 you must assign the same gateway ip 213.xxx.xxx.177 on it as on your host server. Be sure that this ip isn't used somewhere.

In addition:

Clear all iptables rules as described at here

Before doing this the iptables service must be started. Restart the iptables service after clearing to store cleared rules on disk.

edited Apr 13 '17 at 12:14

Community

1

answered Jan 27 '17 at 09:22

Mikhail Khirgiy

2,073
11
7

Thanks, with `213.xxx.xxx.1` as a gateway, I can ping `213.xxx.xxx.1` and `213.xxx.xxx.180` but not `213.xxx.xxx.177` or internet until I stop IPTables - I guess some rules in forward chain are required in order to allow/forward traffic between subnets?! – HTF Jan 27 '17 at 13:28
If your iptables rules are now the same as they was showed hire then you won't have any problem. All is allowed. – Mikhail Khirgiy Jan 27 '17 at 14:20
Can you ping ip address `213.xxx.xxx.1` from internet? – Mikhail Khirgiy Jan 27 '17 at 14:57
Yes, I can ping `213.xxx.xxx.1` from internet but when I ping VM (213.xxx.xxx.2) I got `From 213.xxx.xxx.180 icmp_seq=1 Destination Host Prohibited` so it looks like IPTables are blocking it and yes, rules are the same – HTF Jan 27 '17 at 15:27
Clearing of iptables will help you. – Mikhail Khirgiy Jan 27 '17 at 17:21

score 1 · Accepted Answer · answered Jan 28 '17 at 11:41

Solution which "answers the question":

Make sure ip forwarding is enabled on the host if cat /proc/sys/net/ipv4/ip_forward is not 1, look on this site how to change it
Change the VM gateway to the hosts ip on the vm subnet: 213.xxx.xxx.1
Fix your broken iptable configuration. iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT looks fine. Eventually change it to something link iptables -I FORWARD -s 213.xxx.xxx.0/27 -j ACCEPT ( this means: accept all traffic from the vms network to an other network )

The idea of this solution is to use your host as gateway for the vm network. Obviously if you already have an other gateway on the 213.x.x.0 network ( 213.xxx.xx.180 ) you can use it directly from your vms.

The suggested iptables configuration will allow traffic from the VMs to the outside. If you want to allow connexions from the outside to the VMs you have to put a rule looking like iptables -I FORWARD -d 213.xxx.xxx.0/27 -j ACCEPT

Recommended solution: Keep in mind that all this is probably just a mess because of the way the question is asked. In the real world, if you have a router / firewall which manages your two subnets, you should take the easy way: - the router/firewall will have an ip address on each subnet. It will be gateway for all the hosts in the subnets - assign a vlan to each subnet - on your host, you will have 2 interfaces and to bridges - your vms will be bridged on the "vms subnet" and use the firewall as gateway

This way you will have no crazy iptables / alias / forwarding configuration. You will have some hosts on one subnet, and others on the other subnet, and the routing / acls / nats configured on your firewall. You should not even assign an ip address on the "vm network" to your host.

Merging two networks on the same broadcast domain is a source of problems, and configuring multiple ip addresses on a host should be avoided if you are not very self-confident in network configuration.

Thanks, this is a colo server so I don't have access to network equipment and I have to deal with it as it is. — HTF, Jan 29 '17 at 06:37

KVM bridge over bonding with multiple subnets

2 Answers2