protocol buffers: how to serialize and deserialize multiple messages into a file (c++)?

Question

I am new to Protocol Buffers and c++ but my task requires me to use the two. I want to write a structure of data ( message) into a single file multiple times and be able to read the data. i can read and write a single message but multiple messages is proving harder. I have looked for answers for hours but i can't seem to be able to read the data as a structure. Any example code or pointers will be very helpful.

This is the format of my structure:

    typedef struct Entry
       {
          char  name[ NAME_MAX];
          int   id;
          int   age;
          char  DoB[32]; 
       } entry;

This is what i've been using to write into a file:

    Person_File file;
    fstream output("file", ios::out | ios::trunc | ios::binary);
    file.SerializeToOstream(&output);

I have tried to change the file editing options to append instead of truncate but that does not let me read the data in the format that I want.

this is what I use to read:

    fstream input("file", ios::in | ios::binary);
    file.ParseFromIstream(&input);

These are the contents of my .proto file:

    message Person {


message File {
    required string pname =1;
    required int32 id =2;
    required int32 age =3;
    required string DoB =4;
}

repeated File file =1;

}

From all the searching i've done it seems like CodedInputStream/CodedOutputStream are my best options but I haven't been able to find any detailed examples or explanation to help me understand. I understand that Protocol Buffers are not self delimiting and this is probably the reason why I can't read my messages back in the original format. Any help would be appreciated. Thanks

EDIT: I have tried to use CodedOutputStream based on the messages I've recieved but it doesn't seem to write anything into the file.

    int fd = open("file.txt",O_WRONLY | O_APPEND | O_CREAT);

    FileOutputStream* file_ostream_ = new FileOutputStream(fd);
    CodedOutputStream* ostream_ = new CodedOutputStream(file_ostream_);


ostream_->WriteLittleEndian32(file.ByteSize());
    file.SerializeToCodedStream(ostream_);

After using this code the file comes out blank. Where am I going wrong ?

I don't know about protocol buffers library but I do know you can use `std::fstream.write` to write `Entry` using the `sizeof(Entry)` to the file and read it back using `std::fstream.read`.. It's a POD structure. It shouldn't need any special libraries imo.. — Brandon, Apr 05 '14 at 14:56
possible duplicate of [Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?](http://stackoverflow.com/questions/2340730/are-there-c-equivalents-for-the-protocol-buffers-delimited-i-o-functions-in-ja) — Alan Stokes, Apr 05 '14 at 16:29
The easiest way is to write the size before the message; have a look at http://stackoverflow.com/questions/5586323/storing-multiple-messages-in-one-protocol-buffer-binary-file/5588303#5588303 and https://developers.google.com/protocol-buffers/docs/techniques?csw=1#streaming — Bruce Martin, Apr 05 '14 at 20:23
Thank you for your answers. I have looked at the links and tried to use CodedOutputStream without any luck. I have uploaded the code I used — user3499746, Apr 05 '14 at 21:45

score 5 · Accepted Answer · answered Apr 06 '14 at 20:37

I have tried to use CodedOutputStream based on the messages I've recieved but it doesn't seem to write anything into the file.

I suspect your problem here is that you aren't deleting the CodedOutputStream nor the FileOutputStream. These objects buffer output and flush the buffer in their destructors, so if you never destroy them, they won't write the last buffer, which in this case is the only buffer.

I recommend allocating these objects on the stack (as local variables). Then you can't possibly forget to destroy them.

With that out of the way, here's code that uses CodedInputStream and CodedOutputStream to read/write delimited messages. Each message is prefixed with a varint indicating the size, which is the same format as the Java Protobuf library's writeDelimitedTo()/parseDelimitedFrom() (a feature that never made it into the C++ library).

bool writeDelimitedTo(
    const google::protobuf::MessageLite& message,
    google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
  google::protobuf::io::CodedOutputStream output(rawOutput);

  // Write the size.
  const int size = message.ByteSize();
  output.WriteVarint32(size);

  uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
  if (buffer != NULL) {
    // Optimization:  The message fits in one buffer, so use the faster
    // direct-to-array serialization path.
    message.SerializeWithCachedSizesToArray(buffer);
  } else {
    // Slightly-slower path when the message is multiple buffers.
    message.SerializeWithCachedSizes(&output);
    if (output.HadError()) return false;
  }

  return true;
}

bool readDelimitedFrom(
    google::protobuf::io::ZeroCopyInputStream* rawInput,
    google::protobuf::MessageLite* message) {
  google::protobuf::io::CodedInputStream input(rawInput);

  // Read the size.
  uint32_t size;
  if (!input.ReadVarint32(&size)) return false;

  // Tell the stream not to read beyond that size.
  auto limit = input.PushLimit(size);

  // Parse the message.
  if (!message->MergePartialFromCodedStream(&input)) return false;
  if (!input.ConsumedEntireMessage()) return false;

  // Release the limit.
  input.PopLimit(limit);

  return true;
}

Bingo!!!. Deleting CodedOutputStream and FileOutputStream did the trick. Thank you for your excellent answer — user3499746, Apr 06 '14 at 21:23
I assume `rawOutput` must be initialized with size `message.ByteSize() + sizeof(size)`, right? — kakyo, Nov 20 '19 at 15:33
@kakyo Most stream types don't require specifying a size upfront. If you are specifically serializing into a byte array then I do not recommend using delimited format -- the point of this format is to support writing multiple messages to a stream. But to directly answer your question, no, `message.ByteSize() + sizeof(size)` is not correct. You would need `message.ByteSize() + google::protobuf::io::CodedOutputStream::computeRawVarint32Size(size)`. — Kenton Varda, Nov 21 '19 at 16:59

protocol buffers: how to serialize and deserialize multiple messages into a file (c++)?

1 Answers1

Linked