2 NICs 2 IPS same subnet

Question

I am setting up a SUSE 11 box that has two NICs:

eth0 - 192.168.10.150/24 - 00.0C.29.DC.10.CA (MAC)
eth1 - 192.168.10.151/24 - 00.0C.29.DC.10.DE (MAC)

I need both in the same subnet since they will ne connected to another device (storage) point to point. But I found something while doing the installation; doing ARPs from a workstation I found the behavior below

[root@workstation ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.10.151           ether   00:0C:29:DC:10:CA   C                     eth0
192.168.10.150           ether   00:0C:29:DC:10:CA   C                     eth0

as you can see it's like eth0 gets all the requests. In fact if I disconnect the cable from eth1 IP .151 will be still pingable; which is not what I want. And if I disconnect the cable from eth0 none of the IPs .150 - .151 are pingable, when .151 should still be accessable.

Why is this happening? I need eth0 to bind only to .150 and eth1 to bind only to .151. It seems like eth0 owns the IPs.

This is not working:

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

neither is this

net.ipv4.conf.default.arp_filter=1
net.ipv4.conf.all.arp_filter=1

Any suggestions?

Answer 1

I agree with womble's answer, but I'll provide an answer to your question directly.

First
The 2 settings you mentioned are the ones you want.

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

However after setting this you must force an ARP update. Since you just enabled these settings, all the remote hosts still have the old ARP entries cached and it will appear as if they didnt do anything.

arping -U -c 2 -I eth0 192.168.10.150
arping -U -c 2 -I eth1 192.168.10.151

This will send out an ARP broadcast for each IP updating the cache of any neighbors.

Second
You need to set up source based routing rules.
This gets quite a bit more complicated as routing rules are automatically set up when you bring the interface online. You need to override those rules.

ip route add 192.168.10.0/24 dev eth1 src 192.168.10.151 table 151
ip rule add from 192.168.10.151/32 lookup table 151 prio 10000
ip route del 192.168.10.0/24 dev eth1 src 192.168.10.151 table main

The first line creates a new routing rule to route traffic out the right interface for 192.168.10.151. The second line tells the kernel to use this routing rule if the source IP is 192.168.10.151 (this lookup occurs before the 'main' table because of our priority we assigned it). The third line removes the old rule from the 'main' routing table so traffic going out the other interface doesnt use it accidentally.

Now, youre going to need to set this up to somehow run when your interfaces are brought online. This is distro-specific, so I cannot tell you how to do this part.

Third
Things should work without this, but you might also want to add the following to sysctl:

net.ipv4.conf.eth0.rp_filter=0
net.ipv4.conf.eth1.rp_filter=0

This will keep the kernel from dropping any traffic that comes in the wrong interface.
(UPDATE: Note this is rp_filter, not arp_filter. I think this is the setting you meant to say you already tried)

Answer 2

"I need both in the same subnet since they will ne connected to another device (storage) point to point."

Your rationale makes no sense. If they're both in the same subnet, then by definition it can't be a point-to-point link. If you've got the same IP subnet configured on two different broadcast domains... well, you're screwed. Fix that, and all your problems go away.