8

I have one or more ByteBuffers containing parts of a single message. Now I want to read this message but I do not want to copy N ByteBuffer into a single one. My parser expects a single ByteBuffer with the complete message, but my message is divided into N ByteBuffers.

Is there a way to combine these N ByteBuffers into a single one without byte copying? I imagined some kind of smart implementation of the ByteBuffer abstract class that is backed up by these ByteBuffer under the hood and just adjust the pointers and delegates to the correct ByteBuffer.

In case you are curious why I need that, check the protocol below from BM&F/Bovespa. They break a message into chunks and they can come out-of-order in different packets, in other words, the same message sequence can come in multiple packets, each one with a message chunk. I can't write sequentially to the same ByteBuffer because these chunks can be out of order. :(

Am I missing something smarter here? It looks like there is no way to write sequentially to the same ByteBuffer given this chunk protocol below. :(

enter image description here enter image description here

Michelle Queen
  • 249
  • 3
  • 10
  • 1
    If you don't want to combine all of them into a single buffer, then why does your parser expect a single ByteBuffer? Sounds like a code architecture issue. – jlewkovich Sep 18 '14 at 03:49
  • @JL You might be right, but I don't think so. Check my edit to include the protocol I am trying to implement. – Michelle Queen Sep 18 '14 at 04:51
  • Surely they provide a Java API? You shouldn't have to do all this yourself. FIX APIs for Java are certainly available from other sources. – user207421 Sep 18 '14 at 04:58
  • @EJP They do not provide a UDP client to aggregate packets into a single message. This is the responsibility of the client. I am not concerned about the FIX/FAST part. I am concerned about the low level UDP protocol that needs to reassemble all chunks as they say in the last paragraph above. – Michelle Queen Sep 18 '14 at 05:04
  • OK, well I don't know why you think there is no way to write sequentially to a `ByteBuffer.` That's exactly what all the index-less `put()` methods do. You can even put the contents of one `ByteBuffer` entirely into another one in a single operation. – user207421 Sep 18 '14 at 05:09
  • @EJP You probably did not read the protocol spec above. Let's say message 3231 has 4 parts. And they come in UDP packets out of order, like 0,2,3,1. I can't write that to the same ByteBuffer. I have to somehow reassemble all chunks in the correct order. I think the solution just popped to my mind as I was typing this, but I would like to wait for other people's take on that. – Michelle Queen Sep 18 '14 at 05:14
  • I am not a big fan of ByteBuffer in general so just leaving this link here: https://github.com/nablex/utils-io – nablex Sep 18 '14 at 05:59
  • I addressed a statement that you made. I don't know why you made it, but you made it. – user207421 Sep 18 '14 at 06:05
  • Such a "smart implementation with delegates" would certainly be possible. But the effort for such an implementation would be tremendous, because the `java.nio.*` package/classes are sealed and not designed for being extended: One would have to create dozens of classes in order to offer the full functionality for such a buffer. Additionally, it could not be nearly as efficient as just copying the smaller buffers into a larger one. So unless you urgently *need* such an implementation (e.g. for "write-through" behavior), it's probably not worth the effort, but only a technical gimmick... – Marco13 Sep 18 '14 at 09:40

2 Answers2

0

I think that the data structure you are looking for is called (to my best knowledge) a Chain Buffer. Internally it is a dynamically growing collection (e.g. ArrayList in Java) of byte arrays or buffers, while on the outside the structure acts like a regular Buffer.

It can be quite handy if you dealing with performance, though the implementation of some operations may not be completely trivial (e.g. searching the position of a given pattern).

The only public Java implementation I'm aware of is ChainBuffer in Kraken by NCHOVY (although not maintained anymore).

stuchl4n3k
  • 588
  • 5
  • 14
-4

You can chain ByteBuffers to each other, returning a ByteBuffer as follows:

bba.append(bbb.toBytes()).append(bbc.toBytes())

This results in a single ByteBuffer, which can be passed to your parse method.

It effectively creates a copy, but unless you assign it to something, it will be queued for GC.

If you don't know the count of ByteBuffers, use a List of ByteBuffers, and add each one on the fly, then append them in a loop. You can sort the list before appending so the chunks appear in the order you want.

CharlieS
  • 1,432
  • 1
  • 9
  • 10
  • I can't find this append method on the `ByteBuffer` javadoc. Do you mean the put method? Even if it exists, it looks like it is just copying the bytes to a new ByteBuffer, no? To take the N ByteBuffers and copy all their data to a new ByteBuffer is the trivial and costy solution. Do you agree? – Michelle Queen Sep 18 '14 at 05:39
  • http://download.oracle.com/otn_hosted_doc/jdeveloper/1012/jdev-doc/oracle/ide/util/ByteBuffer.html – CharlieS Sep 18 '14 at 07:04
  • Some people down vote to make their own comments seem more important.. Not my issue and don't care :) – CharlieS Sep 18 '14 at 07:10
  • Copying the data is trivial but cost comes down to many things. I prefer code that is maintainable over efficient. Efficiency can be resolved by cheap resources. Bad code is expensive to maintain. Obviously there is a balance. :) – CharlieS Sep 18 '14 at 07:12
  • 2
    The downvote *may* be due to the fact that this refers to `oracle.ide.util.ByteBuffer`, and not to `java.nio.ByteBuffer`... – Marco13 Sep 18 '14 at 09:26