1

I have multiple CentOS boxes (both Centos 7) with two different servers. These servers have 4 10GB interfaces each. They are connected to two Cisco Nexus switches. Each NIC has a different IP address, all in the same domain. In my example, my server has the following IP addresses on the following NICs:

2: ens5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:12:84 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.203/24 brd 172.20.1.255 scope global ens5f0
       valid_lft forever preferred_lft forever
3: ens5f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:12:85 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.204/24 brd 172.20.1.255 scope global ens5f1
       valid_lft forever preferred_lft forever
4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:06:10 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.200/24 brd 172.20.1.255 scope global ens4f0
       valid_lft forever preferred_lft forever
    inet 172.20.1.201/24 brd 172.20.1.255 scope global secondary ens4f0:1
       valid_lft forever preferred_lft forever
5: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 68:05:ca:2d:06:11 brd ff:ff:ff:ff:ff:ff
    inet 172.20.1.202/24 brd 172.20.1.255 scope global ens4f1
       valid_lft forever preferred_lft forever

That looks pretty normal to me. I have one interface with a secondary IP, but whatever.

When I check the switch, or any other server on the same VLAN as my server, pinging each of the IPs works, however all of the traffic traverses one interface, and the MAC addresses all belong to the one interface carrying traffic:

172.20.1.200    00:01:56  6805.ca2d.0610  Vlan720         
172.20.1.201    00:15:00  6805.ca2d.0610  Vlan720         
172.20.1.202    00:14:56  6805.ca2d.0610  Vlan720         
172.20.1.203    00:14:53  6805.ca2d.0610  Vlan720         
172.20.1.204    00:14:48  6805.ca2d.0610  Vlan720

When I try in the server to ping out of one of the other interfaces, I get no reply.

# ping -I ens5f1 172.20.1.1
PING 172.20.1.1 (172.20.1.1) from 172.20.1.204 ens5f1: 56(84) bytes of data.
^C
--- 172.20.1.1 ping statistics ---
51 packets transmitted, 0 received, 100% packet loss, time 51204ms

As would be obvious, none of the other interfaces move any traffic:

┌nmon─16g─────────────────────Hostname=zfs1─────────Refresh= 2secs ───19:38.03───────────────┐
│ Network I/O ───────────────────────────────────────────────────────────────────────────────│
│I/F Name Recv=KB/s Trans=KB/s packin packout insize outsize Peak->Recv Trans                │
│  ens5f1      0.0       0.0       0.5    0.0    60.0    0.0        0.0      0.0             │
│      lo      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0             │
│  ens4f0    405.4      45.8     337.9  128.5  1228.6  365.0      511.4    144.9             │
│  ens5f0      0.0       0.0       0.5    0.0    60.0    0.0        0.0      0.0             │
│  ens4f1      0.2       0.0       3.0    0.0    61.3    0.0        0.3      0.0             │
│────────────────────────────────────────────────────────────────────────────────────────────│

This is while sending data to one of the other IPs.

I do not have LACP or any other configuration which would give me this expected result. I do not know why the MAC addresses are not being advertised correctly. Any ideas would be appreciated. Multiple servers with different hardware are experiencing this issue. They are also going to two different Cisco Nexus switches with trunk ports between them.

Thanks!

user1955162
  • 296
  • 3
  • 16
  • What you wanted to achieve by assigning several different interfaces to the same network? Linux will be confused on the *routing* level (OSI L3), and MAC addresses have nothing to do with this. – Nikita Kipriyanov Jan 29 '23 at 05:31
  • 2
    Linux uses Weak [Host Model](https://en.wikipedia.org/wiki/Host_model): the IPs belong to the host not to the NIC, and by default it will choose whatever NIC suits its decision algorithm to use an IP address (this can also be related to the NIC that got its ARP reply seen first etc.). You have to set up policy routing to have it behave differently. A Q/A where I made an answer about something similar: https://serverfault.com/questions/900385/ghost-ping-on-a-multi-nic-linux-system – A.B Jan 29 '23 at 12:01
  • This explains it. We moved our multiple interfaces to unique broadcast domains. Thanks! – user1955162 Feb 20 '23 at 08:33

0 Answers0