RFC /advice: On secure/unsecure rpc/event-stream protocol design

Question

SSL seems quite bloated for what I want to do, and I have a passionate hatred for OpenSSL (NSS might be useable). I need to open a TCP channel between two nodes that will be used for RPC / encrypted RPC / Event streams / Encrypted event streams. I am using protocol buffers to define and multiplex the different traffic sources.

I don't want to use SSL to start with. I need authenticated secure key-establishment (authenticated diffie-hellman) and then perhaps a block-cipher based stream object to encypher and decypher encrypted event streams and encrypted RPC.

The first thought that I had was, save coding time and design time by building on an SSL implementation, provided that I can get the socket handle from the SSL implementation and use it for unencrypted exchanges as well as encrypted exchanges. But this will probably end up to be an ugly implementation, and for all I know doing this might be incompatible with the ssl protocol (i.e., strong-coupling between TCP state and SSL state).

The second thought I had was, save coding time and design time by opening multiple sockets. But as we all know multi-socket protocol design is evil.

The third thought was, I'll just encrypt everything, but the service in question serves in the capacity of a high performance event switch and it has a database server running on the same machine as well. The overhead of this approach doesn't satisfy as the majority of traffic will be cleartext.

So, these approaches don't seem satisfactory to me. Therefore, I have come to the conclusion that using cryptopp and boost::asio I can roll my own solution and construct my own protocol (which I already have to do). I am a pretty capable systems programmer and I have an engineers understanding applying encryption techniques.

I am all for reuse, and in this case I wish I could reuse SSL, but I don't think I can. Any advice you can give me from your experience in similar situations (you must have designed or worked on a network protocol) will be appreciated. The advice that makes the biggest impression on me gets the tick.:D

p.s., My application also needs to perform some exotic encryption for which I am pulling in cryptopp anyway.

Answer 1

You certainly can re-use SSL - there is no untoward coupling with TCP state. You can use SSL over any underlying stream you like, the only dependency is that it must be two-way.

The simplest way to do so will be to use an SSL library that allows you to provide your own send/receive functions to be called by the SSL library when it has encrypted-side data to send or receive. You would then implement these functions by wrapping the SSL data within special frames within your own underlying protocol.

(I am only familiar with how the OpenSSL library does this - the SSL_set_bio() function - but I am sure that other SSL implementations allow something similar).

Note though that the computation overhead of the key-agreement part of the protocol - whether done through SSL or roll-your-own - will far outweigh the actual block cipher encryption/decryption, so "encrypt everything" may not be as much of a loss as you expect.