Same network on different interface

Question

We work with devices that are flashed (factory) using TFTP on a usb network connection.

The server has a fixed 192.168.2.100 address and the device a fixed 192.168.2.101 address. When it starts, it connects to download the firmware.

In the current setup, only one device can work at the same time. But I'd like to make it possible to flash as many devices as we can plug (because we can have some massive flashing requirements).

To bypass the routing issue, I made a xinetd version that sets a setsockopt to SO_BIND_DEVICE.

But what I didn't except is that Linux is unable to handle the ARP requests on both interfaces at the same time.

if I do a "ping 192.168.2.101 -I usb0" and "ping 192.168.2.101 -I usb1" at the same time, it will work on one interface:

ARP, Request who-has sk tell 192.168.2.100, length 28
ARP, Reply 192.168.2.101 is-at 7a:0f:66:7c:fc:2c (oui Unknown), length 28
IP 192.168.2.100 > 192.168.2.101: ICMP echo request, id 21807, seq 1, length 64
IP 192.168.2.101 > 192.168.2.100: ICMP echo reply, id 21807, seq 1, length 64

But on the other it won't:

IP 192.168.2.100 > 192.168.2.101: ICMP echo request, id 31071, seq 1, length 64
ARP, Request who-has 192.168.2.100 tell 192.168.2.101, length 28
ARP, Request who-has 192.168.2.100 tell 192.168.2.101, length 28
IP 192.168.2.100 > 192.168.2.101: ICMP echo request, id 31071, seq 2, length 64
IP 192.168.2.100 > 192.168.2.101: ICMP echo request, id 31077, seq 1, length 64
ARP, Request who-has 192.168.2.100 tell 192.168.2.101, length 28

The server doesn't seem to answer to the ARP request.

This is how the connection from the device is handled on the server with a /etc/network/if-up.d/000-first script :

ifconfig $IFACE up
ifconfig $IFACE 192.168.2.100

PID=/var/run/xinetd-$IFACE.pid

# this is the modified xinetd version to bind on one address
kill -9 `cat $PID`
xinetd -pidfile $PID -interface $IFACE

# I tried this to force the handling of ARP table per interface, but it doesn't change anything:
# /usr/sbin/arpd -b /tmp/$IFACE.db -a 3 -k $IFACE 

Here is the modified xinetd version: https://github.com/fclairamb/xinetd/commit/1f5c1e8f9944e372b137e6aa46247f8de807bece#L8R253

Answer 1

First, if you haven't tried proxy arp by itself, you should try proxy arp.

echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp

Option 1: some connmark/policy route mangling

If proxy arp alone didn't work, I can't guarantee this will work, but it's worth a shot, I think. What I'm about to describe is how to set up policy routing to bind responses based on input interface using conntrack. I will describe how to set it up for two interfaces, and it should be easy enough to expand to any number. On the other hand, instead of working, it may simply get mightily confused and not do a damn thing (probably because the kernel arp table doesn't know it should cache ip/mac pairs per device). If that happens, try the second, more ugly method.

First, set up some connmark mangle rules:

iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT

In /etc/iproute2/rt_tables add two tables, 100 and 101, label them usb0 and usb1

100    usb0
101    usb1

For each of these tables add the following (replace <N> with the appropriate number):

ip route add 192.168.2.0/24 dev usb<N> table usb<N>
ip rule add fw <N> table usb<N>
iptables -t mangle -A PREROUTING -i usb<N> -j MARK --set-mark <N>

I'm not certain, but you may need to set up a dummy interface like so for your tftp daemon to listen on:

modprobe dummy
ifconfig dummy0 192.168.2.100/32 up
echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp
sysctl net.ipv4.ip_forward=1
iptables -I FORWARD -s 192.168.2.100 -j ACCEPT
iptables -I FORWARD -d 192.168.2.100 -j ACCEPT

Now test with two devices. If it works, excellent, otherwise try the next method:

Option 2: kvm container per usb net device

Option 2 is uglier, involving some bridging and qemu/kvm. virt-manager is probably the easiest way to create these things.

Create tap devices and bridges, one per usb interface.

tunctl -t tap<N>
brctl addbr usb<N>br
brctl addif usb<N>br tap<N>
brctl addif usb<N>br usb<N>  # this may need to be done each time the usb device is connected/disconnected.

Create a kvm image file or boot CD that is designed to be read only and hosts your tftp server and image... or omit read-only and make one image file per vm (using snapshots for this would be better, but that's more out of scope than this answer is getting).

Run a kvm using the tap interface and your image file and test usb network connectivity.

Comparison

If connmark/policy route worked, it's much easier to synchronously maintain the tftp repository. With KVM, you may need to have a repository per VM image (unless you pass a read-only directory through to each VM, which is doable). Connmark/policy route is a bit more finicky, but it also only requires one tftp server, though if your ports aren't random enough, conntrack could get confused and overwrite if there is a port overlap (or it may not). On the other hand, bridging to kvms requires much more memory; an entire VM per attached USB device, but it's much more likely to work every time because the VM has an entire isolated network stack to work with while the host kernel just has to pass packets back and forth over a bridge, especially true if bridge filtering is off.

Hopefully one of these two work for you.