2

If I defined:

message A {
 required int32 first = 1;
 optional int32 second = 2;
}

The size of space when set_second(0) then serializeToArray() is not same with that when set_second(14353355445) then serializeToArray()

Are there ways to make them have the same size of space ?

In other ways,how can I make optional fields that are not assigned values have the same size of space with just like they were assigned values?

李鹏程
  • 81
  • 1
  • 4
  • Don't use protobuf? This is built into protobuf's purpose. Find some other serialization library. – Dark Falcon Sep 22 '16 at 16:43
  • @DarkFalcon There's nothing wrong with using protobuf as a serialization library. – πάντα ῥεῖ Sep 22 '16 at 16:46
  • 1
    @πάντα ῥεῖ: I didn't say there was. I said if he wants to always include all fields *as asked in the question*, that's probably not going to happen with the stock protobuf library. – Dark Falcon Sep 22 '16 at 16:47

2 Answers2

1

My recommendation is: Don't try to do this with Protobuf. Protobuf is not designed to give you fixed nor predictable sizes, so if that's what you want, then Protobuf isn't the right tool for the job.

Sure, you can use hacks like using fixed32 and trying to make sure your fields always have non-default values, but then you are fighting your tools. Something else could change in the future which makes the sizes unpredictable again. Better to choose a tool that matches your needs.

Kenton Varda
  • 41,353
  • 8
  • 121
  • 105
  • Hey Kenton creator of protobuf. Deep obeisance. The OP sent me a [comment most recently](http://stackoverflow.com/questions/39644416/how-can-optional-fields-that-are-not-assigned-values-in-protocbuf-be-allocated-s/39644611?noredirect=1#comment66607593_39644611) that confirms its a XY problem. IMO protobuf seems to be perfect for them. They need to be as little space used as possible, not fixed size. – πάντα ῥεῖ Sep 23 '16 at 03:59
  • You made my professional in production code running like a charm! Deep obeisance again. – πάντα ῥεῖ Sep 23 '16 at 04:04
  • (I actually suspect OP doesn't need fixed-width at all but it's hard to tell without more details.) – Kenton Varda Sep 23 '16 at 21:26
  • I'd suspect they need as small size as possible, from the information they'd been given so far. But yes, the question and the restrictions are unclear. – πάντα ῥεῖ Sep 23 '16 at 21:30
0

It's encoded in a special way to save bandwidth, hence the different overall size. If you want/need it always the same size you can use the fixed32 field type alternatively.

You should note though, if you don't set a value for the optional field at all, the serialized size will differ again.

πάντα ῥεῖ
  • 1
  • 13
  • 116
  • 190
  • Note that this still does not force an optional field to be present in the message. – Dark Falcon Sep 22 '16 at 16:47
  • @DarkFalcon That's essentially true, but OP mentioned `set_second(0)`. The difference in size comes from the encoding mechanism. – πάντα ῥεῖ Sep 22 '16 at 16:50
  • The OP also says "optional fields that are not assigned values". In any case, as of proto3, even explicitly setting it to the default appears to no longer send it on the wire: "The behavior changed between proto2 and proto3. Under proto3, there is no longer a separate notion of "presence". A field is sent on the wire if and only if it is not equal to its default value." http://stackoverflow.com/questions/9168052/how-do-has-field-methods-relate-to-default-values-in-protobuf – Dark Falcon Sep 22 '16 at 16:56
  • @DarkFalcon I added a note about that to my answer. May be it's a XY problem, because I don' really see why one needs to have always the same size for serialized data. – πάντα ῥεῖ Sep 22 '16 at 16:58
  • @李鹏程 Well, you should appreciate the protobuf facilities to render as little space as necessary actually? I can't share our woes here. – πάντα ῥεῖ Sep 23 '16 at 03:56
  • @πάνταῥεῖ this problem mainly comes from that I have a K-V storge，the size of value is limited 62K, the protobuf message is like message B {repeated message A},message A has optional fields.when I add so many message A which optional fields are not assigned values that the size of value is very cloes to 62k, and when I want to assign optional fields , the size of value will be over 62K – 李鹏程 Sep 23 '16 at 06:13
  • @李鹏程 If you want/need to have smallest possible size, don't set optional values at all, protobuf format is focused to render messages to use a minimum space, either on disk or the wire. – πάντα ῥεῖ Sep 23 '16 at 06:27