1

I am using Ubuntu 16.04.6 LTS and bonded 2 network interfaces with the following config:

auto eno0
iface eno0 inet manual
        bond-master bond0

auto eno1
iface eno1 inet manual
        bond-master bond0

auto bond0
iface bond0 inet static
        bond-slaves none
        bond-miimon 100
        bond-mode balance-xor
       address [IP]
       netmask [IP]
       network [IP]
       broadcast [IP]
       gateway [IP]
       # dns-* options are implemented by the resolvconf package, if installed
       dns-nameservers 8.8.8.8 8.8.4.4

Network is working, but traffic goes almost exclusively only through one interface:

       eno0                eno1               bond0
 KB/s in  KB/s out   KB/s in  KB/s out   KB/s in  KB/s out
 3436.42  100942.1     11.59      0.00   3447.97  100929.5
 3450.31  98191.30     10.32      0.00   3461.29  98208.12
 3426.57  94762.22      9.16      0.00   3436.65  94754.99
 3579.97  101313.8     10.66      0.00   3588.80  101302.2
 4029.16  97184.86      9.93      0.00   4038.42  97196.67
 3225.66  90987.53     13.48      0.00   3239.43  90991.49

When I unplug eno0, traffic moves to eno1. When I plug it back, it goes back to eno0. AFAIK, traffic should be more or less evenly split between the two interfaces since it's generated by thousands of users.

Any ideas what could be wrong?

EDIT: 

cat /sys/class/net/bond0/bonding/mode
balance-xor 2
skall
  • 11
  • 2
  • 6
  • How did you test? By default `balanced-xor` load balances based on MAC address so you would need more than one client to utilize both physical connections. You also could try to set `xmit_hash_policy=layer3+4` which should load balance based on used ports. With `xmit_hash_policy=layer3+4` and `iperf -P 4` should also be able to utilize both links from one client. – Thomas Mar 14 '19 at 17:31
  • It's live traffic generated by thousands of clients = thousands of MAC addresses. – skall Mar 14 '19 at 18:08
  • Can you add the output of `cat /sys/class/net/bond0/bonding/mode` to your question. – Thomas Mar 14 '19 at 18:18
  • I added it to the end. – skall Mar 14 '19 at 20:03
  • 1
    @skall what about switch configuration? – Alexander Tolkachev Mar 14 '19 at 20:13
  • Switch is managed by data center and I don't have access to its config. Anything that could be a problem there? – skall Mar 14 '19 at 21:07

1 Answers1

1

If both eno0 and eno1 are connected to the same switch, or there is some form of interconnection between the two switches. Then you need to configure static aggregation on the switch.

This is because the switch acts as a Layer 2 device, it will automatically records the mapping between MAC addresses and ports (By default, Each MAC address can only be mapped to one port at a time). But in mode 2, all NICs under bond0 share one same MAC address.

If your switch supports LACP, consider using 802.3ad (mode 4). Otherwise, if there is no switch configuration permission, only mode 1, 5 or 6 can be used.

ASBai
  • 141
  • 4