writeDelimitedTo/parseDelimitedFrom seem to be losing data

Question

I am trying to use protocol buffer to record a little market data. Each time I get a quote notification from the market, I take this quote and convert it into a protocol buffers object. Then I call "writeDelimitedTo"

Example of my recorder:

try {
    writeLock.lock();

    LimitOrder serializableQuote = ...
    LimitOrderTransport gpbQuoteRaw = serializableQuote.serialize();
    LimitOrderTransport gpbQuote = LimitOrderTransport.newBuilder(gpbQuoteRaw).build();
    gpbQuote.writeDelimitedTo(fileStream);

    csvWriter1.println(gpbQuote.getIdNumber() + DELIMITER+ gpbQuote.getSymbol() + ...);

} finally {
    writeLock.unlock();
}

The reason for the locking is because quotes coming from different markets are handled by different threads, so I was trying to simplify and "serialize" the logging to the file.

Code that Reads the resulting file:

FileInputStream stream = new FileInputStream(pathToFile);
PrintWriter writer = new PrintWriter("quoteStream6-compare.csv", "UTF-8");

while(LimitOrderTransport.newBuilder().mergeDelimitedFrom(stream)) {
   LimitOrderTransport gpbQuote= LimitOrderTransport.parseDelimitedFrom(stream);

   csvWriter2.println(gpbQuote.getIdNumber()+DELIMITER+ gpbQuote.getSymbol() ...);

}

When I run the recorder, I get a binary file that seems to grow in size. When I use my reader to read from the file I also appear to get a large number of quotes. They are all different and appear correct.

Here's the issue: Many of the quotes appear to be "missing" - Not present when my reader reads from the file.

I tried an experiment with csvWriter1 and csvWriter2. In my writer, I write out a csv file then in my reader I write a second cvs file using the my protobufs file as a source.

The theory is that they should match up. They don't match up. The original csv file contains many more quotes in it than the csv that I generate by reading my protobufs recorded data.

What gives? Am I not using writeDelimitedTo/parseDelimitedFrom correctly?

Thanks!

score 5 · Accepted Answer · answered Dec 20 '13 at 23:49

Your problem is here:

while(LimitOrderTransport.newBuilder().mergeDelimitedFrom(stream)) {
  LimitOrderTransport gpbQuote= LimitOrderTransport.parseDelimitedFrom(stream);

The first line constructs a new LimitOrderTransport.Builder and uses it to parse a message from the stream. Then that builder is discarded.

The second line parses a new message from the same stream, into a new builder.

So you are discarding every other message.

Do this instead:

while (true) {
  LimitOrderTransport gpbQuote = LimitOrderTransport.parseDelimitedFrom(stream);
  if (gpbQuote == null) break;  // EOF

Thank you! I knew there was something a bit weird about that loop. I was looking for some kind of "hasNext" functionality, but I was definitely mistake. Either way that fixed my issues. — eak12913, Dec 22 '13 at 20:55

writeDelimitedTo/parseDelimitedFrom seem to be losing data

1 Answers1