Changing protobuff optional field to oneof

Question

I have the following message:

message Message {
    int64 id = 1;
    google.protobuf.FloatValue weight = 2;
    google.protobuf.FloatValue override_weight = 3;
}

and I wish to change the type of weight and override_weight(optional fields) to google.protobuf.DoubleValue so what I did was the fllowing:

message Message {
    int64 id = 1;
    oneof weight_oneof {
        google.protobuf.FloatValue weight = 2 [deprecated=true];
        google.protobuf.DoubleValue double_weight = 4;
    }
    oneof override_weight_oneof {
        google.protobuf.FloatValue override_weight = 3 [deprecated=true];
        google.protobuf.DoubleValue double_override_weight = 5;
    }
}

My question is, lets assume I have old messages who were compiled by the previous protobuff message compiler for the old message, would I be able to parse them as the new message? The documentation is very vague about this:

"Move optional fields into or out of a oneof: You may lose some of your information (some fields will be cleared) after the message is serialized and parsed. However, you can safely move a single field into a new oneof and may be able to move multiple fields if it is known that only one is ever set."

Has anyone tried this before? what is the best practice for this situation?

Bart · Accepted Answer · 2020-02-05T19:18:55.567

As far as I know fields in an oneof are just serialize using their tag number. The serialized data does not indicate if a field is part of an oneof. This is all handled by the serializer and deserializer. So as long as the tag numbers do not conflict it can be assumed that it will work in both directions, old messages to a new serializer and new messages to an old serializer.

You could test this using an online protobuf deserializer.

Verification: The code does indeed produce the same byte strings. Below you will find the message definitions and python code I used. The python code will output a byte string you can copy and use in the decoder of Marc Gravell.

syntax = "proto3";

message MessageA {
    int64 id = 1;
    float weight = 2;
    float override_weight = 3;
}

message MessageB {
    int64 id = 1;
    oneof weight_oneof {
        float weight = 2 [deprecated=true];
        double double_weight = 4;
    }
    oneof override_weight_oneof {
        float override_weight = 3 [deprecated=true];
        double double_override_weight = 5;
    }
}

import Example_pb2

# Set some data in the original message
msgA = Example_pb2.MessageA()
msgA.id = 1234
msgA.weight = 3.21
msgA.override_weight = 5.43

# Output the serialized bytes in a pretty format
str = 'msgA   = '
for x in msgA.SerializeToString():
    str += "{:02x} ".format(x)
print(str)

# Next set the original fields in the new message
msgB = Example_pb2.MessageB()
msgB.id = 1234
msgB.weight = 3.21
msgB.override_weight = 5.43

# Output the serialized bytes in a pretty format
str = 'msgB 1 = '
for x in msgB.SerializeToString():
    str += "{:02x} ".format(x)
print(str)

# And finally set the new fields in msgB
msgB.double_weight = 3.21
msgB.double_override_weight = 5.43

# Output the serialized bytes in a pretty format
str = 'msgB 2 = '
for x in msgB.SerializeToString():
    str += "{:02x} ".format(x)
print(str)

The output of the python script was:

msgA   = 08 d2 09 15 a4 70 4d 40 1d 8f c2 ad 40
msgB 1 = 08 d2 09 15 a4 70 4d 40 1d 8f c2 ad 40
msgB 2 = 08 d2 09 21 ae 47 e1 7a 14 ae 09 40 29 b8 1e 85 eb 51 b8 15 40

As you can see message A and message B yield the same byte string when setting the original fields. Only when you set the new fields you get a different string.

+1 not for paragraph one: though I assume you're right I have no way to check it; instead, +1 for the link to the online protobuf deserializer! I would never have thought to look for this useful tool and am kicking myself for not thinking of it because I could have used it in the past! (Other tools at the link too.) (Actually, I could check your claim using the online protobuf deserializer, as you yourself suggest, but don't have the time right now ... maybe someone else will report?) — davidbak, Feb 05 '20 at 15:36
@davidbak Good point about the weak first paragraph. I have added some code with proof. — Bart, Feb 05 '20 at 19:19
I would give you a +1 for that improvement but I can't vote up twice for the same answer, sorry! — davidbak, Feb 05 '20 at 21:08
@davidbak No problem I will add it to my private honor account ;-) — Bart, Feb 06 '20 at 07:42

Changing protobuff optional field to oneof

1 Answers1