All interfaces not being used despite multiple sockets

Question

Despite binding two UDP sockets to 2 different addresses on different interfaces of a host, traffic flows through a single interface.

The topology of the network is as follows: 2 links between h2 and s1 1 link between h1 & s1

 ________
|        |
h2       s1_______h1
|________|

I'm emulating 2 hosts and one switch on mininet. h1 is running a UDP server at 10.0.0.1:4243. The other host has 2 interfaces, with ips 10.0.0.2 & 10.0.0.4 . I'm making 2 sockets on h2, binding one to (10.0.0.2,9999) & the other socket to (10.0.0.4,8888). I'm running the code below, which should send packets on both interfaces alternately.

Instead, the first packets are sent on both the interfaces. All subsequent packets are sent over a single interface.

Client Code (Running on h2)

def client():
    sock1 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock1.bind(("10.0.0.2",9999))

    sock2 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock2.bind(("10.0.0.4",8888))

    while True:
        text = 'The time is {}'.format(datetime.now())
        data = text.encode('ascii')

        sock1.sendto(data, ('10.0.0.1', 4243))
        data, address = sock1.recvfrom(MAX_BYTES)

        sock2.sendto(data, ('10.0.0.1', 4243))
        data, address = sock2.recvfrom(MAX_BYTES)

Server Code (Running on h1)

def server():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind(('10.0.0.1', 4243))
    print('Listening at {}'.format(sock.getsockname()))
    while True:
        data, address = sock.recvfrom(MAX_BYTES)
        text = data.decode('ascii')
        print('The client at {} says {!r}'.format(address, text))
        text = 'Your data was {} bytes long'.format(len(data))
        data = text.encode('ascii')
        sock.sendto(data, address)

One packet is sent over both interfaces of h2. Subsequent packets are all sent and received only via interface with IP 10.0.0.2. On wiresharking, half the packets have src/dest IPs set to 10.0.0.4

Answer 1

From your comments, here is what I think is happening:

1. You have conflicting default routes
2. You have ip_forwarding enabled allowing internal routing
3. The bind call is essentially being negated due to #1 and #2

Conflicting default routes

From your comments, both interfaces probably have a /8. This means for h2 the kernel adds the same route for both something like:

10.0.0.0/8 dev h2-eth0 proto kernel scope link src 10.0.0.2 
10.0.0.0/8 dev h2-eth1 proto kernel scope link src 10.0.0.4 

This means the kernel wants to pick only one of these to send all 10.0.0.0/8 traffic to/from. You are trying to get around this with a bind call, but I don't believe Bind will overwrite the routeing table behavior.

ip_forwarding

If this flag is set, it also complicates matters because it may be allowing eth1 traffic via eth0.

bind

So bringing this all together, you bind your sockets to the individual IPs which are associated with individual interfaces, but the kernel only wants to route traffic destined for 10.0.0.1 out one interface and it chooses eth0. I think normally this would prevent eth1 traffic, but due to the routes and the ip_forward flag, eth1 traffic is routed through eth0 (after an initial ARP packet, the one packet you see).

Solution

Give h1's interface two IP addresses 10.1.0.1/16 and 10.0.0.1/16. Then give h2-eth0 ip address 10.0.0.2/16 and h2-eth1 ip address 10.1.0.2/16. Then have your program send data to 10.1.0.1 from 10.1.0.2, and 10.0.0.1 from 10.0.0.2.

By isolating the subnets you will prevent the root cause of the confusion, which was the conflicting routes.