4

I'm working on a low-level UDP messaging layer for an encrypted P2P architecture, if interested you can read more about it on its github page.

I've built a neat serialization framework that turns POJOs into compact ByteBuffers, and also various libraries that make using symmetric and asymmetric crypto fairly painless.

I'm now working on the messaging framework, which makes use of dynamic proxies to achieve similar functionality to GWT's RPC mechanism.

My problem is that early-on I decided to make the serialization mechanism read to and write from ByteBuffers. I'm now finding that this has several problems:

  • You need to know the maximum bytebuffer size in advance of serializing an object
  • They are mutable which makes them error-prone
  • They aren't especially compatible with DatagramPacket, and DatagramChannels are confusing

Can anyone suggest alternative ways to implement serialization in this framework?

sanity
  • 35,347
  • 40
  • 135
  • 226
  • Oh, and if this project interests you - I need the help of solid Java programmers so please get in touch. – sanity Mar 14 '11 at 20:37

2 Answers2

3

Might be worth having a look at KryoNet - it may fit your needs, or alternatively you can take a look at how they do things under the hood.

Incidentally, they do use ByteBuffers for both TCP and UDP connections. I think the trick is to abstract away from ByteBuffers so that any client code can just work with POJOs. To do this, you clearly will need to have logic somewhere that can break a large object into multiple buffered writes.

I think you actually still want ByteBuffers under the hood since they are very fast, support various native acceleration tricks and because like it or not, ultimately you need to deal with buffering in a high throughput IO environment....

mikera
  • 105,238
  • 25
  • 256
  • 415
  • Yeah, I've looked at Kryonet but I don't think it supports the kind of low-level crypto I need, so I have to roll my own. – sanity Mar 14 '11 at 21:33
  • hmmm tricky. always a possibility you could merge your crypto layer into kryonet as some kind of extension? otherwise it just sounds like you will be reinventing lots of wheels..... – mikera Mar 14 '11 at 21:49
  • 1
    Sometimes the only alternative to reinventing the wheel is to hammer a square peg into a round hole (yay mixed metaphor! ;) – sanity Mar 14 '11 at 22:03
0

Can anyone suggest alternative ways to implement serialization in this framework?

Why alternative, here I address your main concerns:

  • You need to know the maximum bytebuffer size in advance of serializing an object

Why max size of ByteBuffer? You can use pool of Buffers. You should be prepared to split the objects into multiple packets. One packet - one object will not work for larger objects. Serialization is interesting point thought: and usually that's where it gets messy. You will need some decent protocol layer (+- resending packets, etc)

  • They are mutable which makes them error-prone

You do want 'em mutable for performance reasons. I have never had issues with them being mutable. You possibly want direct buffers as well. Thank Goodness, that they mutate since they are not cheap to (de)allocate. Direct buffers map straight into the OS code (look at sun.nio.ch.DatagramDispatcher )

  • They aren't especially compatible with DatagramPacket, and DatagramChannels are confusing.

They are ok, you just need to split your objects into multiple packets. If you use buffers, stick to buffers and it should be fine.

bestsss
  • 11,796
  • 3
  • 53
  • 63
  • In general when working with UDP, you want to size your objects such that you're guaranteed to fit inside a single packet. UDP doesn't guarantee delivery or sequence, so you have to architect things with these limitations in mind. I.e. its good for transferring data that is updated frequently (i.e. entity transforms in a game) or that works well with a best-effort update (i.e. chunks in a P2P exchange). – Jolly Roger Feb 28 '13 at 01:12
  • @Jolly, Even for games you need to send the initial state that may be more than megabytes in size, so it won't fit in a single UDP packet, then you need (N)ACK and so on. Of course, UDP has its limitations, no guarantees so picking it over TCP should be well reasoned. – bestsss Feb 28 '13 at 12:22