3

using rapidJSON I am able to store data of any type (supported by rapidJSON) as value against a key. Can I do that in protobuf? If so, how?

The reason I opted for protobuf over rapidJSON is because of its speed (and key comparison in rapidJSON is really a "string" comparison which is costly). Or is there any alternative to that? maybe flatbuffers?

Requirement is storing data of any type against an int type key. Faster serialization/deserialization than regular json.

Looking for answers from C++ perspective.

helix
  • 1,017
  • 3
  • 12
  • 30

2 Answers2

1

In protobuf, typically you have a fixed contract that pre-interprets given keys as given types, but: there is also the concept of extensions. With an extension, arbitrary data can be stored against field-numbers; this works for any type that could also have been expressed using the regular API.

The convenience and performance of the extension API depends on the implementation, but it should be perfectly usable from the official C++ API.

The key point about extensions is that only the consumer needs to understand them.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
1

Both Protobuf and FlatBuffer have a dictionary feature (see https://developers.google.com/protocol-buffers/docs/proto#maps and https://google.github.io/flatbuffers/md__cpp_usage.html under dictionaries). The big problem you may have with both however is not convenient to have the value be an arbitrary value, since both are defined by a schema, meaning you have to specify an actual type for the value. You can get around that by defining unions of all possible types, but it is never as convenient as JSON.

FlatBuffers however has a dedicated format for storing any value without a schema: https://google.github.io/flatbuffers/flexbuffers.html. This is a lot faster than JSON, more compact, and uses less extra memory to read (none).

FlatBuffers has the ability to use an int as key, but FlexBuffers doesn't yet, so you could consider storing a FlexBuffer as value inside a FlatBuffer int dictionary.

Both format parse from JSON and output to JSON, even when nested.

FlexBuffers can't be modified in-place. FlatBuffers can, using its object API. So again nesting could work well as long as you're ok re-generating the entire FlexBuffer value when it changes.

A final alternative worth mentioning is a std::map<int, std::vector<uint8_t>> (or unordered_map) to store a map of FlexBuffers directly. That is simpler, but now the problem you have is not having a convenient way to store the whole thing.

Aardappel
  • 5,559
  • 1
  • 19
  • 22