0

In my app, I must respond to a request with a list of results as protocol buffers. The list could be very long, which means it could eat up a lot of memory for both the sender and the receiver if it were all kept in memory at once.

One option is to paginate, but that requires keeping track of pages. I'd rather stream it in chunked encoding. I'd like the sender to be able to serialize a chunk of data, send it and forget it, and I'd like the receiver to be able to receive some data, de-serialize it and process it, without either side ever having everything in memory.

My question is, how can the client know where one protocol buffer ends and the next one begins? Is this information inside a protocol buffer, or do I need to create some kind of delimiting scheme myself?

Nick Retallack
  • 18,986
  • 17
  • 92
  • 114

1 Answers1

0

There is no information about the size of the message inside a protocol buffer message itself. So, you'd need to add a length prefix to each chunk yourself.

One common way to do it is to add length prefix encoded using varint encoding. See this answer for details on how this can be done in C++.

Community
  • 1
  • 1
alavrik
  • 2,151
  • 12
  • 16