Protobuf concatenation of serialized messages into one file

Question

I have some serialization in google protobuf in a series of files, but wonder if there is a shortcut way of concatenating these smaller files into one larger protobuf without worrying about reading each and every protobuf and then grouping these objects and outputting.

Is there a cheap way to join files together? I.e. do I have serialize each individual file?

score 3 · Answer 1 · answered Nov 27 '13 at 19:32

3

You can combine protocol buffers messages by simple concatenation. It appears that you want the result to form an array, so you'll need to serialize each individual file as an array itself:

message MyItem {
   ...
}

message MyCollection {
   repeated MyItem items = 1;
}

Now if you serialize each file as a MyCollection and then concatenate them (just put the raw binary data together), the resulting file can be read as a one large collection itself.

answered Nov 27 '13 at 19:32

jpa

10,351
1
28
45

5

To be precise, concatenating encoded protobufs and then parsing the result is equivalent to if you parsed them separately and did `first.MergeFrom(second)`. – Kenton Varda Nov 28 '13 at 06:13

score 1 · Accepted Answer · edited May 23 '17 at 12:05

In addition to jpas answer, it might be relevant to say that the data does not need to be in the exact same container, when being serialized, for it being compatible on deserialisation.

Consider the following messages:

message FileData{
    required uint32 versionNumber = 1;
    repeated Data initialData = 2;
}

message MoreData{
    repeated Data data = 2;
}

It is possible to serialize those different messages into one single data container and deserialize it as one single FileData message, as long as the FileData is serialized before zero or more MoreData and both, the FileData and MoreData have the same index for the repeated field.

Protobuf concatenation of serialized messages into one file

2 Answers2