2

I am using a TCP Client/Server to send Cap'n Proto messages from C++ to Java.

Sometimes the receiving buffer may be overfilled or underfilled and to handle these cases we need to know the message size.

When I check the size of the buffer in Java I get 208 bytes, however calling 

MyModel.MyMessage.STRUCT_SIZE.total()

returns 4 (not sure what unit of measure is being used here).

I notice that 4 divides into 208, 52 times. But I don't know of a significant conversion factor using 52.

How do I check the message size in Java?

BAR
  • 15,909
  • 27
  • 97
  • 185

1 Answers1

2

MyMessage.STRUCT_SIZE represents the constant size of that struct itself (measured in 8-byte words), but if the struct contains non-trivial fields (like Text, Data, List, or other structs) then those take up space too, and the amount of space they take is not constant (e.g. Text will take space according to how long the string is).

Generally you should try to let Cap'n Proto directly write to / read from the appropriate ByteChannels, so that you don't have to keep track of sizes yourself. However, if you really must compute the size of a message ahead of time, you could do so with something like:

ByteBuffer[] segments = message.getSegmentsForOutput();
int total = (segments.length / 2 + 1) * 8;  // segment table
for (ByteBuffer segment: segments) {
  total += segment.remaining();
}
// now `total` is the total number of bytes that will be
// written when the message is serialized.

On the C++ size, you can use capnp::computeSerializedSizeInWords() from serialize.h (and multiply by 8).

But again, you really should structure your code to avoid this, by using the methods of org.capnproto.Serialize with streaming I/O.

Kenton Varda
  • 41,353
  • 8
  • 121
  • 105
  • In this case I have to split my ByteStrings to feed to Serialize.read so I need to know the message size. I am building the messages in C++ and reading in Java so I don't think your code would be appropriate for this. Are the messages the same fixed length if not using the non-trivial fields? – BAR Aug 15 '15 at 23:36
  • Why can't you pass the underlying stream to Serialize.read()? Why are you reading into ByteStings first? (And what is a ByteString? The class from Protobuf?) – Kenton Varda Aug 16 '15 at 02:36
  • I am using `akka.util.ByteString` pretty much like the Akka server example. I use `byteString.asByteBuffer` when passing to `Serialize.read()`. – BAR Aug 16 '15 at 02:55
  • It would really help if I could ask `message` how many bytes it read. – BAR Aug 16 '15 at 03:24
  • @BAR `read()` will advance the `ByteBuffer`'s position to the end of the message it reads. So you can find out how many bytes were read by checking the position before and after. – Kenton Varda Aug 16 '15 at 04:15
  • I actually tried just that but I am getting 8 for position. I am trying to make sense of this. – BAR Aug 16 '15 at 04:16
  • I also realized there is a problem if the message is incomplete. Serialize possibly waits for the data if incomplete but in my case the ByteBuffer is an immutable slice of a ByteString. It is the ByteString that will be updated with the rest of the data. – BAR Aug 16 '15 at 05:59
  • I can now confirm the unit size used in `ByteString` and `ByteBuffer` are the same. So I think there might be a bug because each message is 208 bytes in size however `byteBuffer.position` returns only 8 bytes after reading. – BAR Aug 16 '15 at 23:01
  • Oh hmm, maybe the ByteBuffer position isn't properly updated. I'd consider that a bug. You should file it on the github repo. You should not pass an incomplete messages as a ByteBuffer; if the message may not be complete you should write a stream or channel implementation to represent it. – Kenton Varda Aug 16 '15 at 23:47
  • That seems to be it. But even if it worked I am going to need to write the message size before the message. I think this will be speedier than converting/wrapping a ByteString to a stream. Could you please update your answer to include finding size on the C++ side? Opened issue: https://github.com/dwrensha/capnproto-java/issues/37 – BAR Aug 17 '15 at 00:00
  • 1
    In C++ use `capnp::computeSerializedSizeInWords()` (updated answer). – Kenton Varda Aug 17 '15 at 01:35