We are using two switches (let's call them A and B), and each host (running debian 9) has 2 nics (eno1 and eno2), connected to A and B. Those interfaces are bonded together (bond0) in active-backup mode:
# ip link show dev eno1
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 7a:6a:2c:d8:83:82 brd ff:ff:ff:ff:ff:ff
# ip link show dev eno2
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 7a:6a:2c:d8:83:82 brd ff:ff:ff:ff:ff:ff
# ethtool -P eno1
Permanent address: ac:16:2d:72:75:14
# ethtool -P eno2
Permanent address: ac:16:2d:72:75:15
# cat /etc/modprobe.d/bonding.conf
options bonding max_bonds=0 miimon=100 mode=active-backup
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eno2
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eno2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:16:2d:72:75:15
Slave queue ID: 0
Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:16:2d:72:75:14
Slave queue ID: 0
Over this stack, we use VLANs: 72 is the "production" vlan (private IPs) and 73 is the "public" one. Those VLANs are bridged inside two bridges: brprod and brpub. That way, we can just set tap in the proper bridge to get virtual machines' traffic on the appropriate vlan.
All this is configured using Systemd-network:
# cat 10-all-nic-to-bond0.network
[Match]
Name=eno[1-2]
[Network]
Bond=bond0
# cat 20-bond0.netdev
[NetDev]
Description=Underlying bonding
Name=bond0
Kind=bond
[Bond]
Mode=active-backup
MIIMonitorSec=1s
# cat 20-bond0.network
[Match]
Name=bond0
[Network]
VLAN=prod
VLAN=public
LinkLocalAddressing=no
BindCarrier=eno1 eno2
# cat 30-vlan-prod.netdev
[NetDev]
Name=prod
Kind=vlan
[VLAN]
Id=72
# cat 30-vlan-prod.network
[Match]
Name=prod
[Network]
Bridge=brprod
# cat 30-vlan-pub.netdev
[NetDev]
Name=public
Kind=vlan
[VLAN]
Id=73
# cat 30-vlan-pub.network
[Match]
Name=public
[Network]
Bridge=brpub
# cat 40-brprod.netdev
[NetDev]
Name=brprod
Kind=bridge
# cat 40-brprod.network
[Match]
Name=brprod
[Network]
DHCP=no
Address=10.0.0.143/24
# cat 40-brpub.netdev
[NetDev]
Name=brpub
Kind=bridge
# cat 40-brpub.network
[Match]
Name=brpud
[Network]
DHCP=no
After adding a public IP to the pub vlan interface (to spare me from starting a VM), I can't reach that address. What bothers me is that production network does work; that's how I'm connected to the host.
# networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eno1 ether carrier configuring
3 eno2 ether carrier configuring
4 eno3 ether off unmanaged
5 eno4 ether off unmanaged
6 brprod ether routable configured
21 tap0 ether degraded unmanaged
22 brpub ether degraded unmanaged
25 veth1 ether degraded unmanaged
26 bond0 ether carrier configuring
27 public ether routable configuring
28 prod ether carrier configuring
12 links listed.
After some digging, it looks like some traffic is coming from one nic, and some other from another nic. But the bond doesn't get the sum of it!
# tcpdump -eni eno1 arp and host 80.67.160.69
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
22:10:09.976118 aa:00:00:11:01:f8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 73, p 0, ethertype ARP, Request who-has 80.67.160.69 tell 80.67.160.65, length 46
22:10:10.434247 aa:00:00:b6:73:8e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 73, p 0, ethertype ARP, Request who-has 80.67.160.69 tell 80.67.160.77, length 46
22:10:10.972490 aa:00:00:11:01:f8 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 73, p 0, ethertype ARP, Request who-has 80.67.160.69 tell 80.67.160.65, length 46
^C
3 packets captured
4 packets received by filter
0 packets dropped by kernel
# tcpdump -eni eno2 arp and host 80.67.160.69
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
# tcpdump -eni bond0 arp and host 80.67.160.69
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Unordered questions:
- Why would
ip linkandethtool -Preturn a distinct mac address? - Do you have any remark on the systemd-networkd configuration files? I ran into the "numbered" filenames because I couldn't find how the files get read.
- Why would the bond not get the aggregation of both interfaces?
Thanks for reading thus far!
