I have several physical network interfaces on a debian 6 box, each has a different subnet IP address. Two of the interfaces are connected to a switch hub so they can communicate with each other. I want to setup a TCP connection between the 2 interfaces.
This is what I do: I first get a SOCK_STREAM socket. I then use setsockopt(..., SO_BINDTODEVICE,...) to bind the socket to a specific interface, eth4. I then setup a sockaddr_in structure and populate it with the port and address of that interface (eth4). I then call bind() to bind that local address/port to the socket. I then do a listen() followed by an accept(). That blocks waiting for an incoming connection request. At this point I can do a netstat and it shows exactly what I expect, that is, the socket is indeed in LISTEN state and the Local Address show the eth4 address and port correctly.
Then in a separate program, I have a routine that gets a socket, then setsockopt(..., SO_BINDTODEVICE,...) to bind the socket to a different interface, eth5. I then setup a sockaddr_in structure using the same port number I used for the eth4 interface, and the address of the eth4 interface. I then do a connect(). All goes well, the connect succeeds and my logging indeed shows that the 2 processes are indeed connected with this TCP connection.
But then when I do a netstat, it shows that the 2 PID/Programs as being connected, but both the Local Address and the Foreign address show the same address from eth4 even though I specifically bound the connect end to the address of eth5.
So, my question is why does netstat show the same address for both sides of the connection?
The only explanation I can come up with is that the kernel recognizes that the connect destination address is actually local and then ignores the SO_BINDTODEVICE binding I did and makes the TCP connection internal instead of actually going out eth5 to connect to eth4.
If that is correct, how do I then actually make a TCP connection between 2 separate interfaces, on the same linux box, but actually going external instead of internal?
-Andres