How to encode multiple distinguishable items back to back with protobuf?

Question

So, I have a unit test where I'm trying to ensure that encoding and decoding is sound. Basically, I kind of want to be able to serialize multiple different C++ objects into a string, and then write that string into a file. This is what I have so far:

Here's my .proto file:

message Register {
    required string regname = 1;
    required uint64 regvalue = 2;
}

message RegisterContext {
    repeated Register elem = 1;
}

message MemoryRegion {
    repeated bytes elem = 1;
}

message Image {
    required string imgName = 1;
    required MemoryRegion Memory = 2;
    required uint64 MemStart = 3;
    required uint64 MemEnd = 4;
    required bool isMain = 5;
}

message MemoryRegions {
    repeated Image elem = 1;
}

message MemoryDump {
    required RegisterContext context = 1;
    required MemoryRegions state = 2;
}

And I have language vernacular style representation objects that help me use them as I would normally:

//name, value
using reg=std::tuple<std::string, uint64_t>;
using reg_context=std::list<reg >;
using mem_region=std::string;
//name, memory, start, end, ismain
using image=std::tuple<std::string, mem_region, uint64_t, uint64_t, bool>;
using mem_region_set=std::list<image >;
using memory_dump=std::tuple<reg_context, mem_region_set>;

Here's the failing test:

TEST(SerializationTests, test_contiguous_memory_differentiated) {
  mem_region region="\x25\x26\x27";
  image i1{std::make_tuple(std::string("img1"), region, 0, 1, false)},
    i2{std::make_tuple(std::string("img2"), region+"\x28", 0, 1, false)},
      i3{std::make_tuple(std::string("img3"), region+"\x29", 0, 1, false)};
  mem_region_set regions1{i1, i2, i3}, regions2{i1, i3};
  auto serialized_regions1=serialize_memory_regions(regions1);
  auto serialized_regions2=serialize_memory_regions(regions2);
  std::stringstream ss;
  ss << *serialized_regions1 << *serialized_regions2;
  auto concatenated=ss.str();
  auto deserialized_region1=unpack_memory_regions(concatenated);
  ASSERT_EQ(deserialized_region1, regions1);
  auto deserialized_region2=unpack_memory_regions(concatenated);
  ASSERT_EQ(deserialized_region2, regions2);
}

How do I serialize multiple objects so that, whether I put them into a string stream or a ofstream, they come back out as being distict? The test that I run here results in deserialized_region1/2 each having 5 total images, merely i1, i2, i3, i1, i3

I don't understand what your question has do do with protobuf? You are trying to do de-/serialization yourself, instead of using the generated c++ classes for your `.proto` files? — πάντα ῥεῖ, Feb 12 '15 at 13:17
No, I don't want to do it myself, I thought that I could store two different protobuf objects back to back do that I can retrieve the first object and the second object from the same file. Do I have to define a new protobuf type for that? — Adam Miller, Feb 12 '15 at 13:24
Where in your code do you invoke the `protobuf` library? I only see that you are serializing these classes by hand with `stringstream`. — Siyuan Ren, Feb 12 '15 at 14:27

score 1 · Accepted Answer · answered Feb 12 '15 at 14:11

One possible way to do it is just to write fixed 32 bytes integer size before each protobuf object. When you need to parse the buffer or file with multiple objects you can first read message length, and then read message body that is length bytes long and parse it. For example your message MemoryDump will be compiled to class MemoryDump. You can pack it like this:

MemoryDump memoryDump;
fillData(memoryDump); // put data in it somehow
auto size = memoryDump.ByteSize(); // return space that it needs to be packed
std::string buffer;
StringOutputStream stream(&buffer);
CodedOutputStream coded(&stream);
coded.WriteLittleEndian32(size);
memoryDump.SerializeToCodedStream(&coded);
coded.WriteLittleEndian32(size);
memoryDump.SerializeToCodedStream(&coded);

And then when you unpack data you can do:

MemoryDump memoryDump1;
MemoryDump memoryDump2;
std::string buffer;
readBuffer(buffer);
StringInputStream stream(&buffer);
CodedInputStream coded(&stream);
int size;
coded.ReadLittleEndian32(&size);
memoryDump1.ParseFromBoundedZeroCopyStream(&stream, size);
coded.ReadLittleEndian32(&size);
memoryDump1.ParseFromBoundedZeroCopyStream(&stream, size);

See also http://stackoverflow.com/a/22927149/2686899 for better-optimized versions of this code. — Kenton Varda, Feb 13 '15 at 04:25

How to encode multiple distinguishable items back to back with protobuf?

1 Answers1