I've 2 Linux boxes with following description
Box A: Kernel: Linux 2.6 IP: 172.19.1.99/24
Box B: Kernel: Linux 3.14 IP: 192.168.200.100/24
Connected with Layer 2 switch
Both boxes run a UDP broadcast service and a UDP Receiver service. Whatever sent by broadcast is received by receiving service. These service together form custom discover protocol used by a Web App running on these boxes.
Problem:
When Box A is moved to 192.168.200.0/24 subnet and then later moved back to 172.19.1.0/24 subnet the receiver service on Box B will still receive broadcast from the Box A but technically it should stop receiving the broadcast from A. Now same service running on Box A does not receive the broadcast from Box B at this point.
Just to confirm I attached strace to the receiving service on Box B and I can see the recvfrom system call is indeed receiving broadcast sent by Box A
to address 192.168.200.255.
Now if I kill the receiving service on Box B and restart it, then it will stop receiving broadcasts sent from Box A.
In short the problem is seen only when a box running in same subnet is migrated to different subnet and is only seen on Box B.
Any idea why this might be happening?.
UPDATE: Just to add more information Box B also uses iptables with conntrack modules can this some how influence this behavior?.
