4

What is the correct way to setup a bonded bridge on Centos 6 for KVM guests?

I'd currently playing around with a setup of two KVM-hosts which will each host several guests. I have two HP DL380:s with 4 nics each. I'd like to use two nics (eth0, eth1) in a active-backup (mode=1) bond, for failover reasons, facing internet. Then i'd like to have the two other nics (eth2, eth3) also in a active-backup bond, facing a admin/back net.

On top of the bond I need a bridge that the KVM guests will use to access front or back network.

On the interwebz I have found many different ways to configure this. Some are just mentioning bonding, some just bridging and some are tring to combine it. None that I have found has metioned what will happen if I use a front and back net with many hosts.

Some of my problems/questions are.

  • I got a fetich on correct config files, the way the developers thought they should be, not just working config files.
  • I got error "kernel: bond0: received packet with own address as source address". Both for bond0 and bond1.
  • Will the traffic automtically be forwarded from the back-net to the front-net. Should I use ebtables/iptables or something to disable the forwarding traffic?
  • Do I need to use Spanning Tree Protocol (STP)?
  • Do I need any specific routes?

Here is a nice picture how the environment looks (at least a part of it.)

Network schema

Here are my relevant config files.

/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=host1
GATEWAYDEV=br0
NETWORKING_IPV6=no

/etc/sysconfig/network-scripts/ifcfg-eth0 -- ifcfg-eth3

DEVICE="ethX"
NM_CONTROLLED="no"
ONBOOT=yes
HWADDR=xx:xx:xx:xx:xx:xx
SLAVE=yes
MASTER=bondX
HOTPLUG=no
BOOTPROTO=none

/etc/sysconfig/network-scripts/ifcfg-bond0 -- ifcfg-bond1

DEVICE=bondX
BONDING_OPTS="miimon=100 mode=1"
ONPARENT=yes
BOOTPROTO=none
BRIDGE=brX

/etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=none

/etc/sysconfig/network-scripts/ifcfg-br1

DEVICE=br1
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=static
IPADDR=10.0.1.100
NETMASK=255.255.255.0

Update 1

  • Added /etc/sysctl.conf
  • Removed ip from ifcfg-br0. The host shouldn't be accessible from internet, only from admin net.

* Update 2*

  • Removed changes to /etc/sysctl.conf. Don't need to enable iptables.
Arlukin
  • 1,203
  • 6
  • 18
  • 27

2 Answers2

2

Not sure about CentOS 6, but on Fedora the bonding module is not added to the Linux kernel by default and therefore you need to create a file /etc/modprobe.d/bonding.conf with content

alias bond0 bonding

Reboot, and you should see bonding module loaded during boot.

Since you have two bonded interfaces you might have to add another alias line for bond1 as well. However I have never tried that.

Suggest you get one working and then worry about setting up the second.

Other issue you raised about the bridge, other points to note, these configurations work with the network daemon but I don't believe they work with NetworkManager. Are you running the network or the NetworkManager daemon?

And finally, there are different ways to configure netfilter to handle bridged interfaces. At least on Fedora 12+, the default is to disable netfilter on bridges. However you can change this by editing /etc/sysctl.conf and setting 

net.bridge.bridge-nf-call-iptables=1

Do the same for arp and ipv6 and in this file you also need to set

net.ipv4.ip_forward=1

Flush your FORWARD chain and replace with iptables rule

iptables -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
interoperate
  • 41
  • 2
  • Looks like "alias bond0 bonding" is not necessary. When looking into /etc/sysconfig/network-scripts/network-functions their is a check for BONDING_OPTS in the config files that installs bonding driver if not already installed. – Arlukin Sep 29 '11 at 11:09
  • Thanks for the tips about ***"net.bridge.bridge-nf-call-iptables"***, this wasn't needed in Centos 5. So saved me from some problems. But is ***"net.ipv4.ip_forward=1"** really necessary? I don't have that on my KVM hosts that are running without bonding, and only one bridge connected to one interface. – Arlukin Sep 29 '11 at 11:09
  • I'm not running the NetworkManager, this is a minmal centos installation without any X. – Arlukin Sep 29 '11 at 11:10
  • What does the output of iptables -L -v show? Especially relevant is the traffic in the FORWARD Chain. – interoperate Sep 30 '11 at 23:40
  • At the moment nothing, accepting everything now when testing. And I have this setting ** net.bridge.bridge-nf-call-iptables=0**. **br0** facing the Internet doesn't have any ip, so I shouldn't need to be afraid that something is coming in that way. – Arlukin Oct 02 '11 at 06:56
2
  1. forwarding to iptables will work, but what for? this will definitely create extra overhead
  2. the networking schema seems right, though I'd try and get the hardware to use mode 4 instead of 1 to be on the safe side, and to also aggregate the traffic
  3. not having an IP addres on br0 and br1, do you have other NICs you can access the host by? You really don't want to lock yourself out (unless you also have an iLO of course)
  4. the kernel message you posted is usually related to using bonding modes incompatible with bridging (0 and 6). Can you make sure youre actually at mode-1 (/proc/net/bonding/...) usually contains an indication
  5. forwarding between subnets should not happen without specific configuration, however, the hosts' kernel should be able to determine which packet is destined where, in br0 and br1 are on different subnets, so a VM on br0 should be able to talk on the br0 network, and if you want it to also be able to communicate on br1, add another virtual NIC and attach it there
dyasny
  • 18,802
  • 6
  • 49
  • 64
  • 1. Probably not needed, if things work as you describe in your point 5. ---- 2. Do I need to do any special configurations in the switch to get mode 4 to work? It's a Cisco Linksys SRW2024 and should support 802.3 ---- 3. I have 4 nics + ILO port that might be possible to use as a regular nic. I have already removed ip from br0 (the front), making it impossible/harder to reach the host from internet. But I can't see the reason not having an ip on br1 on the admin-net. Because the ILO port would also be connected to the admin-net. – Arlukin Sep 29 '11 at 21:22
  • Looks like I need to configure the switch ports for 802.3, atleast regarding to this http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding. Which was a good page explaining bonding. – Arlukin Sep 29 '11 at 22:17
  • 1
    mode 4 requires switch config for link aggregation; no ip on br0 might hinder routing, so I suggest you make sure everything works this way, and if not, then try and configure the routes manually; basically, you've been on the right track all along, I'm pretty sure you could have made things work on your own :) – dyasny Sep 30 '11 at 07:44
  • I will do some tests with setting up the ILO port to also work as an interface. Otherwise I might remove the bonding from the admin-net. Probably not needed there (but it's cool to have in both ends). And you are right, I was almost done with the setup when asking the question here. Just wanted to start to be a part of the serverfault community =) – Arlukin Sep 30 '11 at 11:12