10

Question

I want to connect two processes via TCP but I don't want have to specify which of them is the server and which is the client but they know the IP and host of each other. They should decide on their own which is the server and which is the client and then initiating the connection.

Background

I am working on a bidirectional distributed framework where - in contrast to RPC - there is no client/server model. Instead the distributed components should just be able to talk to each other by specifying a host and port.

Edit: The concept goes beyond the implementation details of a socket connection. This should be a new concept to simplify designing a distributed application in terms of software engineering. This is in contrast to RPC and SOA (which are Server/Client oriented) and message oriented systems (which demand the usage of IMO un-intuitive patterns).

The solution must not

  • defining a protocol via UDP, because I need the TCP reliability and the possibility of SSL usage
  • using a framework like ZeroMQ, because I cannot use binary packages on my target platform
  • edit: a global message broker / name server because it should be a lightweight solution without an additional process. And adding such a node will just reintroduce the client/server concept

Update

After the discussion there seems only one useful approach: Every peer need to have a listing socket (or you cannot do any auto-discovery of course). On a connection request the node will try to connect to the other peer if there isn't already an open connection.

This could be problematic if the connection is done simultaneous, so we will end up with two connections between two peers. The question is now how to deal with that in an async context. I don't think this is as easy as said below in comments because we need to guarantee that only one connection is closed. I think a protocol like 2PC is needed for this task.

schlamar
  • 9,238
  • 3
  • 38
  • 76
  • 1
    `Instead the distributed components should just be able to talk to each other by specifying a host and port.`: that's how normal TCP sockets works, one `bind()` and `listen()`, other `connect()`, i guess you only need to make your program a client+server. The real problem is how peers discover each other without a "server". – KurzedMetal Jun 06 '12 at 13:11
  • @KurzedMetal The point/clue of the concept is that there should be no client/server model. The components are equivalent and pass messages bidirectional. So specifying a client or a server is unnecessary boilerplate or should be at least... – schlamar Jun 06 '12 at 13:15
  • then why would you say `talk to each other specifying a host and port`? what do you actually mean by that?. That's all you need to make a TCP connection, a host and a port. – KurzedMetal Jun 06 '12 at 13:16
  • I need to locate my components and I think there is no other way than by host/port. (Because I don't want to have a global message broker / name server) – schlamar Jun 06 '12 at 13:20
  • Perhaps the "just finding each other" part could be done with [IP multicast](http://en.wikipedia.org/wiki/IP_Multicast)? – Lauritz V. Thaulow Jun 06 '12 at 13:29
  • 3
    I really don't understand you, you say you don't want a client/server model but you are hinting for one everywhere... `one is going to listen and the other is initiating the connection.`... `components should just be able to talk to each other by specifying a host and port`... With those statements you are describing a client/server model. I guess i'll just move on, GL with your project. – KurzedMetal Jun 06 '12 at 13:29
  • 1
    @KurzedMetal You have to look from a higher level in terms of [software engineering](http://en.wikipedia.org/wiki/Software_engineering) to understand my concept. The client/server model is relevant at the step of designing/modelling a software. And in this step I don't want to define any client or server because the nodes should be equal. But in the end the framework should create a TCP connection which needs a node to be listing and the other to start connecting. So the framework should auto-define/discover who is client and who is server. – schlamar Jun 06 '12 at 13:41
  • Software can't auto-discover anything unless something tells it where to look for what in which fashion – Alex Jun 06 '12 at 13:42
  • @alex Yes, the whole point is how to do this in an easy und sensible way – schlamar Jun 06 '12 at 14:02
  • Updated the question so it should be more clear. – schlamar Jun 06 '12 at 14:09
  • You want your peers to connect without knowing _anything_ about each other? Or do they have any information about each other (e.g. IP addresses)? How would your peers even decide to connect in the first place? – lanzz Jun 06 '12 at 14:22
  • @lanzz They know `(ip,host)` and should connect on request. – schlamar Jun 06 '12 at 14:27
  • 2
    On whose request? The peer who is requested to connect to the other would need to have some mechanism to signal that intent to the other peer first, before they start negotiations who is going to listen and who is going to connect. It seems to me like overcomplicating a relatively simple situation; all peers could listen at some port all the time, and the one that is requested to connect to another simply connects on that known listening port; from that point on you have a symmetric connection and it is entirely up to you who will be client and who will be server. – lanzz Jun 06 '12 at 14:32
  • @alex Yes, it can. The one that wants to connect to the other is the client, and the other that does not want to connect to anything is the server. It is possible that they both decide to connect to each other at the same time, at which point it is trivial to detect that they have two open connections, and do a random roll and a losing connection is dropped. – lanzz Jun 06 '12 at 14:40
  • The connection request comes from the process if he wants to talk with the other one. If the request is simultaneous on both nodes there will be two connections with this approach. An alternative solution would be to fix this double connection problem. – schlamar Jun 06 '12 at 14:40
  • @lanzz How is this trivial (in an async context)? – schlamar Jun 06 '12 at 14:41
  • I am in a same situation where I have two threads one which has the server code gets blocked by the listen statement. – Abhinav Oct 23 '12 at 14:39
  • @sHoM That's a completely different problem. Actually, it isn't a problem at all because it is well defined behavior. Calling listen on a socket will usually block. If you make a new question with some code and link it here, I (and others) can help you further. – schlamar Oct 25 '12 at 06:24
  • @schlamar Here is the [**main question**](http://stackoverflow.com/questions/12619068/network-bridge-using-scapy-and-python) and this is [**part i am stuck at**](http://stackoverflow.com/questions/13048950/netdisturb-using-python) – Abhinav Oct 25 '12 at 08:50
  • The question is clear, and is a valid one. You have two peer servers, and you want a single TCP connection between them. The question is, which one initiates the connection? My strategy is to have globally unique ids for each server, and for every server to know the ids of every other server. The one with the higher id is responsible for initiating the connection. – Sriram Srinivasan Jul 20 '13 at 02:25

2 Answers2

8

It appears to me you are slightly confused. Yes, software engineering literature talks about the server/client model and contrasts it with other models, such as peer-to-peer. However, at their core, distributed systems always end up using a server/client model somewhere, because there really is no other way to communicate through the Internet.

(Technically, you should be able to send any kind of IP datagrams through the Internet, so you could attempt to invent a different transport protocol that isn't server/client at its core, but I'm assuming you do not want to install new transport layers into the networking stacks of the OSes, plus, there's other gotchas such as home NAT appliances that might be in the middle that would thwart such communication.)

How can the two nodes "decide between themselves" who is going to listen and who is going to connect if you're restricting all kind of communication from the get go? You can only reliably use two protocols through the Internet: UDP and TCP. Both of them involve a process setting up a listening socket and the other sending a message (UDP) or performing a connection attempt (TCP) to said listening server.

Your idea of doing UDP messages before establishing the connection doesn't really change anything. A process listening to UDP messages is still a server. If anything, it just means that additionally to either listening to or initiating a TCP connection, all the nodes will have to be UDP listening servers as well.

You will need to have a third-party message broker of some kind if you do not want to have any kind of listening sockets in the peers before the handshaking happens. This is obvious, because you can't communicate without listening sockets, so you can't perform said handshake to begin with. Catch-22.

There's also the obvious solution: make each peer listen to connections, then when two peers want to talk to each other, make both attempt to connect to each other: if neither can connect, report that neither peer is able to function as a server. If one can connect, proceed as normal. If both can connect, discard the second connection and continue using the first connection.

My suggestion would be for you to look into how protocols like BitTorrent function (they function quite similarly to the solution I detailed above). Optionally, after you've figured this out, you might want to look into NAT traversal solutions such as STUN to ensure you don't run into the "neither peer can be a server" situation as often, but that's a separate issue.

mpontes
  • 2,936
  • 1
  • 20
  • 22
  • Yes and no ;) I know that I have to use the client/server concept on the core, but the framework should hide this completely to the end user (which other approaches like RPC, Webservices do not). But you are right latter part, I never thought my idea (which came while writing the question) so far that I need a listing socket on each peer to do a auto discovery of client/server :) So the real problem is how to solve this "double connection problem" which seems not this easy for me in an async context. – schlamar Jun 06 '12 at 14:57
  • Doesn't seem particularly hard to solve to me. Make each peer be identified by some kind of ID (unique ID, hostname:port pair, doesn't matter) and have each node keep state on which nodes they already have connections to. Note that, like @Mike-Steder is talking about, the real problem here is peer discovery, not connecting two peers once they both know about each other. It seems to me you're looking at this problem with the wrong assumptions. If you're assuming every node knows about all the other nodes right from the beginning, it's not much of a distributed system. – mpontes Jun 06 '12 at 15:11
  • You just have to be a little careful so you don't end up dropping both connections on a race condition. I suggest some kind of handshaking protocol that goes like this: "Hello, we're going to use this connection" "OK, acknowledged" "Acknoledged." Or "Going to use this" "No, I already have a connection to you, can I drop this connection?" "Yes, you can drop/No, the other connection hasn't finished handshaking for me. Wait a bit." – mpontes Jun 06 '12 at 15:11
  • 1: Why shouldn't it be a distributed system? It is just not P2P. 2: A reliable handshake is not this easy with an asynchronous core. – schlamar Jun 06 '12 at 15:47
2

If you want two peers to find each other you're going to either have to look into either:

  • a broadcast scheme: zeromq would work but if you can't use that you can try to use ip multicasting
  • your own discovery service

Bootstrapping peer-to-peer applications is going to require one of the above. Most end up using the web to "broadcast" a Bittorrent Tracker URL and that then helps peers find one another.

Given your requirements I'd strongly encourage you to look into IP Multicast. I used Twisted (http://twistedmatrix.com/documents/current/core/howto/udp.html) to create a simple UDP protocol. I used this to start up a master process on my desktop machine and multiple worker processes on a set of remote machines to coordinate a load test of a site. I won't say it's trivial but it works quite well.

The key is that you must define your own simple wire protocol, possible something as simple netstring ( + ). I would use a simple structured message format like JSON for easy serialization and to allow you to easily encode metadata in your messages.

You will have a simpler time if you use 2 UDP Multicast Ports, one for discovery and the other for messages or data that you want to send between nodes.

I hope that helps but without more details about what you actually want to accomplish it's difficult to be much more specific.

stderr
  • 8,567
  • 1
  • 34
  • 50
  • No, the point is that I don't want to find the peers. Two peers should decide among themselves which one is going to listing on the socket and which is connecting to this socket. – schlamar Jun 06 '12 at 13:47
  • And how do those peers find each other to decide? If you want to have them elect a leader that's great but they first need to be able to discover each other. – stderr Jun 06 '12 at 14:54
  • By specifying ip/port as pointed out in the question. – schlamar Jun 06 '12 at 14:59