9

On a centos 7 virtual box, I create a team like this:

nmcli connection add type team con-name team0  ifname veteam0
nmcli connection modify team0 team.config roundrobin.conf
[root@rhce1 ~]# cat roundrobin.conf
{
        "device":               "team0",
        "runner":               {"name": "roundrobin"},
        "ports":                {"enp0s9": {}, "enp0s10": {}}
}
# add slave interfaces
nmcli connection add type team-slave con-name team0-port0 ifname enp0s9 master team0
nmcli connection add type team-slave con-name team0-port1 ifname enp0s10 master team0

I now have the following connections:

[root@rhce1 ~]# nmcli con s
NAME                UUID                                  TYPE            DEVICE
team0               77869010-af1a-48a5-b10b-c05b3035837f  team            veteam0
team0-port1         99f54013-b20b-41d2-9661-d654b89870bd  802-3-ethernet  enp0s10
team0-port0         3f5e5924-f7f3-409a-b4cb-661904ddbf60  802-3-ethernet  enp0s9
Wired connection 1  acc2747d-9576-4ac5-a06a-d45d3a9a4ff4  802-3-ethernet  enp0s3
enp0s3              b4db7dd8-8735-4590-b12f-621d1003841a  802-3-ethernet  --
hostonly            8993c4c0-fa90-455d-ae83-d1a644c36886  802-3-ethernet  enp0s8

This brings all interfaces up correctly:

[root@rhce1 ~]# nmcli dev status
DEVICE   TYPE      STATE      CONNECTION
enp0s10  ethernet  connected  team0-port1
enp0s3   ethernet  connected  Wired connection 1
enp0s8   ethernet  connected  hostonly
enp0s9   ethernet  connected  team0-port0
veteam0  team      connected  team0
lo       loopback  unmanaged  --

and I get an IP from dhcp for my virtual team interface:

[root@rhce1 ~]# ip addr show veteam0
6: veteam0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 08:00:27:c7:15:f1 brd ff:ff:ff:ff:ff:ff
    inet 10.23.23.104/24 brd 10.23.23.255 scope global dynamic veteam0
       valid_lft 1073sec preferred_lft 1073sec
    inet6 fe80::a00:27ff:fec7:15f1/64 scope link tentative dadfailed
       valid_lft forever preferred_lft forever

(I am aware that in a typical teaming use case, dhcp is probably not being used, but the main problem also exists with static IPs.)

Now I would like to be able to take the connection down, and later up again, without reboot:

[root@rhce1 ~]# nmcli con down team0
[root@rhce1 ~]# nmcli dev status
DEVICE   TYPE      STATE         CONNECTION
enp0s3   ethernet  connected     Wired connection 1
enp0s8   ethernet  connected     hostonly
enp0s10  ethernet  disconnected  --
enp0s9   ethernet  disconnected  --
lo       loopback  unmanaged     --
[root@rhce1 ~]# nmcli con up team0
Error: Device 'veteam0' is waiting for slaves before proceeding with activation.
[root@rhce1 ~]# nmcli dev status
DEVICE   TYPE      STATE                                  CONNECTION
enp0s3   ethernet  connected                              Wired connection 1
enp0s8   ethernet  connected                              hostonly
veteam0  team      connecting (getting IP configuration)  team0
enp0s10  ethernet  disconnected                           --
enp0s9   ethernet  unavailable                            --
lo       loopback  unmanaged                              -

Now can get the team interface working half way, by connecting enp0s10 first:

[root@rhce1 ~]# nmcli dev connect enp0s10
Device 'enp0s10' successfully activated with '99f54013-b20b-41d2-9661-d654b89870bd'.
[root@rhce1 ~]# nmcli con up team0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

But my team now just consists of one device:

[root@rhce1 ~]# nmcli dev st
DEVICE   TYPE      STATE        CONNECTION
enp0s10  ethernet  connected    team0-port1
enp0s3   ethernet  connected    Wired connection 1
enp0s8   ethernet  connected    hostonly
veteam0  team      connected    team0
enp0s9   ethernet  unavailable  --
lo       loopback  unmanaged    --

My problem is, I currently do not see how to bring enp0s9 up without a reboot, since the device is in state "unavailable". Is my understanding correct, that I should be able to bring enp0s9 up without a reboot, just like enp0s10? To me it looks like there is something fishy with my device enp0s9, but I do not quite see what. I already tried assigning new MAC addresses, but no help. Also, using traditional ifcfg configuration, I can bring my team0 interface up and down without a problem.

Or could it be that my virtual switch (I am using the VirtualBox hostonly network) does not support (? - or needs to be configured to support) teaming?

EDIT: I tried building a team with just one slave. That worked, and I got the same behavior using enp0s9 and enp0s10 - I had to connect the device first bevor I could up the connection. Does this imply my virtual switch is disconnecting one device (which by chance is enp0s9 - but because its deterministic its always enp0s9)?

I don't really need to "Solve" this - I would be happy to give the bounty to someone who helps me understand what is going on.

Isaac
  • 1,215
  • 3
  • 26
  • 44

1 Answers1

1

We using bonding/teaming quite a lot and principally on Oracle Unbreakable Linux (RH CLONE). We typically use NMTUI because we are lazy... but what I noticed in what you are doing is you:

  1. take down the team interface
  2. then bring up the team interface and strangely it is waiting for slaves...
  3. then you manually bring up one of the slaves and the other becomes unavailable.

We don't have to do that. If we down the team interface, when we bring it up it goes through and checks connectivity on the slaves and activates them without any further commands. You never actually "downed" any of the slaves so you shouldn't have to "up" them either.

Did you wait long enough for the slaves to come up? Ours come up almost immediately and I have this on many servers include vsphere on bare metal as well as Oracle Unbreakable Linux.

The "bonded" (team) interface should be managing all the bonding members (slaves). And yes, you can use it with DHCP as well.

Some output examples:

[root@goliath etc]# nmcli connection show
NAME                   UUID                                  TYPE            DEVICE 
Ethernet connection 1  d676e7bb-4612-4946-b5c9-6afd694ec754  802-3-ethernet  em1    
Ethernet connection 3  8c6b99d1-e8c0-494d-9795-4fec4f5ba871  802-3-ethernet  em3    
Bond connection 1      1e7952cf-2b62-4494-8b3c-2b7bdde786c8  bond            bond0  
Ethernet connection 4  7c79016c-95a8-4e19-9a9b-30ea20d5455c  802-3-ethernet  em4    
Ethernet connection 2  b8c82461-9fae-4912-adc8-fc40fb13125b  802-3-ethernet  em2    
[root@goliath etc]# 
[root@goliath etc]# nmcli dev status
DEVICE  TYPE      STATE      CONNECTION            
bond0   bond      connected  Bond connection 1     
em1     ethernet  connected  Ethernet connection 1 
em2     ethernet  connected  Ethernet connection 2 
em3     ethernet  connected  Ethernet connection 3 
em4     ethernet  connected  Ethernet connection 4 
lo      loopback  unmanaged  --                    
[root@goliath etc]# 

And here's what it looks like from nmtui:

NMTUI text-GUI Initial screen

And here's what the interface definition looks like in NMTUI: bonded slave configuration

I also got data from one of the other servers and so maybe this will help:

    ]# nmcli dev show
    GENERAL.DEVICE:                         bond0
    GENERAL.TYPE:                           bond
    GENERAL.HWADDR:                         80:18:44:E5:39:E8
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     etherchannel 13 on CISCO
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/4
    IP4.ADDRESS[1]:                         192.168.10.10/0
    IP4.GATEWAY:                            192.168.10.129
    IP4.ROUTE[1]:                           dst = 192.168.10.129/32, nh = 0.0.0.0, mt = 300
    IP4.ROUTE[2]:                           dst = 192.168.10.10/32, nh = 0.0.0.0, mt = 0
    IP4.ROUTE[3]:                           dst = 192.168.10.129/32, nh = 0.0.0.0, mt = 0
    IP4.ROUTE[4]:                           dst = 0.0.0.0/0, nh = 192.168.10.129, mt = 0
    IP4.ROUTE[5]:                           dst = 0.0.0.0/0, nh = 192.168.10.129, mt = 300
    IP4.DNS[1]:                             192.168.10.203
    IP4.DNS[2]:                             10.1.8.9
    IP6.ADDRESS[1]:                         fe80::8218:44ff:fee5:39e8/64
    IP6.GATEWAY:                            --
    IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
    IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256
    
    GENERAL.DEVICE:                         em2
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         80:18:44:E5:39:E8
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     Ethernet connection 2
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/3
    WIRED-PROPERTIES.CARRIER:               on
    IP4.GATEWAY:                            --
    IP6.GATEWAY:                            --
    
    GENERAL.DEVICE:                         em3
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         80:18:44:E5:39:E8
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     Ethernet connection 3
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1
    WIRED-PROPERTIES.CARRIER:               on
    IP4.GATEWAY:                            --
    IP6.GATEWAY:                            --

    GENERAL.DEVICE:                         em4
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         80:18:44:E5:39:E8
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          100 (connected)
    GENERAL.CONNECTION:                     Ethernet connection 4
    GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/2
    WIRED-PROPERTIES.CARRIER:               on
    IP4.GATEWAY:                            --
    IP6.GATEWAY:                            --
    
    GENERAL.DEVICE:                         em1
    GENERAL.TYPE:                           ethernet
    GENERAL.HWADDR:                         80:18:44:E5:39:EA
    GENERAL.MTU:                            1500
    GENERAL.STATE:                          30 (disconnected)
    GENERAL.CONNECTION:                     --
    GENERAL.CON-PATH:                       --
    WIRED-PROPERTIES.CARRIER:               on

    GENERAL.DEVICE:                         lo
    GENERAL.TYPE:                           loopback
    GENERAL.HWADDR:                         00:00:00:00:00:00
    GENERAL.MTU:                            65536
    GENERAL.STATE:                          10 (unmanaged)
    GENERAL.CONNECTION:                     --
    GENERAL.CON-PATH:                       --
    IP4.ADDRESS[1]:                         127.0.0.1/8
    IP4.GATEWAY:                            --
    IP6.ADDRESS[1]:                         ::1/128

    [root@loki xyz]# nmcli dev status
    DEVICE  TYPE      STATE         CONNECTION              
    bond0   bond      connected     etherchannel 13 on CISCO 
    em2     ethernet  connected     Ethernet connection 2   
    em3     ethernet  connected     Ethernet connection 3   
    em4     ethernet  connected     Ethernet connection 4   
    em1     ethernet  disconnected  --                      
    lo      loopback  unmanaged     --  

The hardest part I had initially was making sure the bonding configuration was correct between the servers and the network switches so both sides would understand the bond. Then we tested by unplugging cables.

I really hope this helped.

Thanks, David

TekOps
  • 71
  • 4