45

I'm trying to upgrade to using protobuf version 3, and stay backwards compatible with version 2. Seems to work except for one thing - in proto-2 you could set your own default values, but in proto 3, you can't. If you chose a default value in proto-2 that is not the standard default value in proto-3, then you have a problem. For example, in proto-2:

message Record {
  required uint32 fileno = 1;               
  required uint64 pos = 2;                  
  optional uint64 bmsPos = 3 [default = 0]; 
  optional uint32 scanMode = 4 [default = 9999];  
}

now in proto-3 must be:

message Record {
  uint32 fileno = 1;               
  uint64 pos = 2;                  
  uint64 bmsPos = 3; 
  uint32 scanMode = 4;  
}

In both proto-2 and proto-3, missing values aren't sent in the message. But the proto-3 API doesn't tell you if the default value is in the message or not, it just tells you the value.

So the proto-3 receiver gets a message and tells me that scanMode = 0. If that message came from a proto-2 sender, then either 1) the proto-2 sender placed a 0 in the message, or 2) the proto-2 sender set the value to 9999 (the default value), and so the value is not sent, and the proto-3 receiver interprets it as a 0. Without knowing if the value is present in the message or not, my code can't disambiguate, even if it knows whether the message came from a proto-2 or proto-3 sender.

Note that there's no problem with the bmsPos field in the example, since the proto-2 message uses the same default value as proto-3 (0). But if you happened to have chosen a default value not the same as proto-3, then I don't see how to upgrade to proto-3 and be backwards compatible.

Null
  • 1,950
  • 9
  • 30
  • 33
John Caron
  • 1,367
  • 1
  • 10
  • 15

1 Answers1

48

Turns out there is a way to find out if a default value is actually missing or not (thanks to some friends at google for this answer):

message Record {
  uint32 fileno = 1;               
  uint64 pos = 2;                  
  uint64 bmsPos = 3; 
  oneof scanMode_present {
    uint32 scanMode = 4;
  }
  uint32 version = 5; // set to >= 3 for protobuf 3 
}

The generate code has additional methods to detect if oneof fields are set, using the getXXXcase() method:

int scanMode = proto.getScanMode();
boolean isMissing = proto.getScanModePresentCase() == Record.ScanModePresentCase.SCANMODEPRESENT_NOT_SET;
if (isMissing) {
  boolean isProto3 = proto.getVersion() >= 3;
  scanMode = (isProto3) ? 0 : 9999;
}
  • Note that the name of the oneof is arbitrary, i have adopted the convention fieldname_present.
  • The oneof doesnt add anything to the wire format, so it stays compatible with the proto-2 messages.
  • You can add the version info anywhere that makes sense, i put it in the Record message for this example.

With this 'trick', i have upgraded to proto-3 with backwards compatibility with non-standard proto-2 default values.

John Caron
  • 1,367
  • 1
  • 10
  • 15
  • 32
    Nice trick! I'm the author of proto2 (but not proto3) and I designed the "oneof" feature, but I'm not sure if I would have thought of this! – Kenton Varda Oct 19 '15 at 07:00
  • 1
    Dont know who figured out the trick. A friend from google tracked it down for me. Thought others will need it. – John Caron Oct 19 '15 at 20:03
  • 1
    @JohnCaron, I was looking for the same, If I understand your solution correcly it works, but I am a bit scared of having to add to all clients the logic to set default values. – Fernando Gonzalez Sanchez Feb 17 '16 at 01:57
  • So basically the oneof is a proto3 version of truly optional field (e.g. it could be not set) and one may detect whether it was set or not. – Paweł Szczur Dec 30 '16 at 10:32
  • From version 3.15 protobuf 3 [officially supports](https://github.com/protocolbuffers/protobuf/blob/main/docs/implementing_proto3_presence.md) `optional` keyword, which is implemented on top of `oneof` under the hood. – Alex Che May 27 '23 at 10:21