Store protobuf byte type inside a binary file

Question

I'm trying to split and combine a binary file, for a reason not related to this question i'm using protobuf to store the files char* in a protobuf byte type

The code to serialize char* looks like this:

char* buffer = new char[buffer_size];
ifstream fileStream;

fileStream.open(fileName,ios::in | ios::binary);

//stuff here

Data *data = new Data(); // Protobuf Constructor

fileStream.read(buffer, buffer_size);
data->set_payload(buffer);
data->set_piece_no(piece_no);

.proto file :

 syntax = "proto3";
 message Data {
 int32 piece_no = 1;
 bytes payload = 2;
 }

Then i try to combine all the pieces like so :

ofstream fileOutput;
fileOutput.open("out.bin", ios::out | ios::binary);
fileOutput << data->payload();

But sadly this doesn't work and the binary file generated is substantially smaller than the original.

I then suspect that the bytes could have null characters \0, and as a result the bytes have actually been truncated.

To test out my hypothesis i do the following:

Data *data = new Header();
data->set_payload("hel\0lo");
data->set_piece_no(piece_no);

ofstream fileOutput;
fileOutput.open("out.bin",ios::out | ios::binary);
fileOutput << data->payload();

Opening the binary file in a text editor (vscode) shows the following:

hel

But the following code:

string data("hel\0lo",6);
ofstream fileOutput;
fileOutput.open("out.bin", ios::out | ios::binary);
fileOutput << data;

Shows the following:

hel?lo

How can i output exactly what i inputted into protobuf, without any truncation because of arbitrary null bytes ?

What if you use `std::ostream.write(...)` instead? I assume you know the length. If not, you'll need to save it somewhere as `strlen` isn't going to work with data that has embedded 0 bytes. — Retired Ninja, May 17 '19 at 22:43

score 1 · Accepted Answer · answered May 19 '19 at 15:08

If you pass a string literal, then it will treat it as such and only read until the first null terminator.

Instead you can pass a std::string directly as in your last example.

See under "Singular String Fields (proto3)" in https://developers.google.com/protocol-buffers/docs/reference/cpp-generated#oneof

Store protobuf byte type inside a binary file

1 Answers1