2

I'm writing an application that has a number (hundreds) of concurrent network operations running in parallel between two instances. Since the average lifetime of a connection is quite short (at most seconds), I thought that the overhead of using many TCP connections and doing a handshake every time (especially for a TLS handshake) would be too large.

I started to look at a couple of protocols and libraries implementing multiplexing (mostly AMQP implementations like Apache Qupid, RabbitMQ, as mentioned in answers to this question). However all of them seem to run over TCP, which introduces some overhead and doesn't make a lot of sense (this post explains the problem quite well and comes to the conclusion that TCP multiplexing is stupid). Also all of them feel quite fat, I'd prefer something small and light (ZeroMQ unfortunately doesn't implement multiplexing afaik). That got me thinking whether using UDP would be an option. Of course one has to implement stuff like recovery and ACKs properly, but with the knowledge about the multiple streams over the connection that should be much more efficient than simply using TCP.

Do you think that my reasoning above is correct, or did I miss something important? Are there any good C/C++ libraries out there that implement multiplexing via UDP?

Community
  • 1
  • 1
lucas clemente
  • 6,255
  • 8
  • 41
  • 61
  • For high-performance network event handling (multiplexing) you could look at either [libevent](http://libevent.org/) or [libev](http://software.schmorp.de/pkg/libev.html). – Some programmer dude Nov 06 '12 at 13:23
  • From the looks of it, you'd be reimplementing TCP in terms of UDP, and I don't see how you'd make it more efficient. The post you link to doesn't really say that 'TCP multiplexing is stupid', more that there are complex risks (eg. local buffers overflowing), and these would tend to apply to simulated TCP as well. – Kylotan Nov 06 '12 at 13:31
  • @Kylotan True, but my "reimplementation" would know about the multiplexed stream. A problem highlighted in the link is that the ACK needed for the multiplexed stream triggers yet another TCP ACK (and therefore adds a roundtrip). That's unneeded overhead that wouldn't happen on UDP. – lucas clemente Nov 06 '12 at 13:34
  • I'll add that you can tune a ack/retry algorithm to the needs of your application in UDP whereas in TCP you cannot (to the same degree). – mark Nov 06 '12 at 13:37
  • Wait, how does the multiplexed stream need an ack which triggers another ack? – Useless Nov 06 '12 at 13:39
  • @Useless See http://www.250bpm.com/multiplexing, point 3 under "Sanity check". The ACK sent by the multiplexing implementation is yet another TCP packet. – lucas clemente Nov 06 '12 at 13:41
  • TCP multiplexing is not stupid!:) Sure, TCP has overhead but most of the overhead is there to ensure that packets arrive in the correct order, that transmissions are reliable, flow control, etc. With UDP you've got to take care of all of that yourself - see your overhead take off :) I'd use TCP multiplexing :) – Anthill Nov 06 '12 at 13:42
  • @lucasclemente that assumes you need a per-logical-stream pushback/flow control mechanism. Besides, these logical acks get Nagled like everything else, so probably add negligible overhead to a _busy_ connection. – Useless Nov 06 '12 at 13:44
  • TCP ACKs are sent as part of the normal dataflow, not as separate packets. (Which I see is what Useless just said.) – Kylotan Nov 06 '12 at 13:44
  • I see, so you're saying that the TCP overhead is negligible? Do you know of any "light" libraries that do multiplexing via TCP? – lucas clemente Nov 06 '12 at 13:51

1 Answers1

5

Do the simplest thing that could possibly work, and make it more complex only as necessary:

  1. use a single TCP connection and multiplex logical sessions over it

    • if these logical entities are just asynchronous request/response pairs, for example, you may be able to dispense with explicit logical sessions entirely

  2. if you have multiple concurrent components in each instance which really need their own queues with pushback to throttle over-eager senders:

    • first consider just capping the number of outstanding requests/active sessions on the sending side, instead of requiring a specific ack
    • only if you need to dynamically vary queue length (eg. because you're really trying to limit working memory, which varies by session) use an explicit logical ack for this

  3. only if you hit some case where your logical sessions really interact badly with TCP, then consider implementing your own reliable-flow-controlled-datagram protocol

Useless
  • 64,155
  • 6
  • 88
  • 132