0

I take a piece of data and put it in a protobuf in both GO and JS and then encode the PB on each platform and the resulting serialized values differ. Since we use the encoded value for signing and hashing, it's critical they match. From what I could gather, I think the difference is the JS is including default values in the encoded output whereas GO is not.

Both platforms start from the same piece of JSON. Here is GO:

listing := new(pb.Listing)
err = jsonpb.UnmarshalString(string(jsonListing), listing)
ser, err := proto.Marshal(listing)
sEnc := b64.StdEncoding.EncodeToString(ser)

the base64 encode value from GO is:

ChR0ZXN0LXRlc3QtdGVzdC1taWxseRLEAQouUW1RMlRoQkw2emNZeEJzQ0gyZlVWM0VVaVBNM3RZbWRuUDNxN2prZTIxdUJNcBpJCiQIARIg1JRiC99XTy49u47TrmPhebH2IoWanvr9rfG2+cj8O4YSIQLJRSKlbpvhAB4nyf2yr0gTbVTXwn8uL41usco/cwtyliJHMEUCIQDqnEyTrFKKNY0FRlbn9wC4+69ozF8C3meKcLQG36nseQIgfJs1dJdFTSM2lGg7hQ68O1PVjAZHWO2XRaogo3OMeUgaKwgEIgYI4LSc/wcqA0xUQyoDQlRDKgNCQ0gqA1pFQzIDVVNEQLgIUIDC1y8ioAIKFFRFU1QgVEVTVCBURVNUIG1pbGx5IGQ6/gEKDG11cmFrYW1pLmpwZxIuUW1WeVZIOFJhbTZNZTNpaHlLZ2p6SnNNMlhaeG5QajZQS1NmbmVSRmY4WmFhRBouUW1laFFoMlNDeVZuWXpZNTduVFAzOWRrbUU3Z0t5ekpHeUhUTko0dXpDM2QyRCIuUW1YcTFSTEt0d2E3VmNSemFhN0dTWEtoVWdIYnBicUhNZWhVS2RDeVVTV1hvNyouUW1VeHlBdHYzdzgxWVFnaEFtckVHTThpbjRYU01QNkROZEVnY1RqNm12UXRjMzIuUW1TODhUcVgySzlwU1VvdnFjczNXbkdhUDFRQjdoTXNSUHdMZFVXNmR5UzRoTFIDTkVXYgAqKQoMVVNBIHNoaXBzdGVyEAEaAuoBKhMKCFN0YW5kYXJkEBkaAzUtNyAK

Here's what JS does:

const ListingPB = getProtoContractsRoot().lookupType('Listing');
const listingPB = ListingPB.fromObject(jsonListing);
const ser = ListingPB
  .encode(listingPB)
  .finish();

Which results in:

ChR0ZXN0LXRlc3QtdGVzdC1taWxseRLGAQouUW1RMlRoQkw2emNZeEJzQ0gyZlVWM0VVaVBNM3RZbWRuUDNxN2prZTIxdUJNcBIAGkkKJAgBEiDUlGIL31dPLj27jtOuY+F5sfYihZqe+v2t8bb5yPw7hhIhAslFIqVum+EAHifJ/bKvSBNtVNfCfy4vjW6xyj9zC3KWIkcwRQIhAOqcTJOsUoo1jQVGVuf3ALj7r2jMXwLeZ4pwtAbfqex5AiB8mzV0l0VNIzaUaDuFDrw7U9WMBkdY7ZdFqiCjc4x5SBo6CAQQABgAIggI4LSc/wcQACoDTFRDKgNCVEMqA0JDSCoDWkVDMgNVU0Q6AEC4CEoAUIDC1y9dAAAAACKxAgoUVEVTVCBURVNUIFRFU1QgbWlsbHkSABoAIGQoADr+AQoMbXVyYWthbWkuanBnEi5RbVZ5Vkg4UmFtNk1lM2loeUtnanpKc00yWFp4blBqNlBLU2ZuZVJGZjhaYWFEGi5RbWVoUWgyU0N5Vm5Zelk1N25UUDM5ZGttRTdnS3l6Skd5SFROSjR1ekMzZDJEIi5RbVhxMVJMS3R3YTdWY1J6YWE3R1NYS2hVZ0hicGJxSE1laFVLZEN5VVNXWG83Ki5RbVV4eUF0djN3ODFZUWdoQW1yRUdNOGluNFhTTVA2RE5kRWdjVGo2bXZRdGMzMi5RbVM4OFRxWDJLOXBTVW92cWNzM1duR2FQMVFCN2hNc1JQd0xkVVc2ZHlTNGhMTQAAAABSA05FV2IGEgAYACAAKikKDFVTQSBzaGlwc3RlchABGgLqASoTCghTdGFuZGFyZBAZGgM1LTcgCkoAUgA=

...which is different than what GO is coming up with.

If I bring both those base64 strings into JS, decode them into a PB and then toJSON() that PB and look at the diff of the two objects, it looks like the difference is that JS is serializing the default values and GO is not (JS is on the right).

enter image description here

I have tried serializing JS as so, but the result is the same:

    const ser =
      ListingPB
        .encode(ListingPB.toObject(listingPB, { defaults: false }))
        .finish();

So, is there any way I could make the output consistent between the two platforms? The raw JSON input is the same going in, but the results on the way out differ.

robmisio
  • 1,066
  • 2
  • 12
  • 20

1 Answers1

0

Ok, this isn't the ideal solution, but a not so pretty work around until the real solution comes, which will probably require a code change to the protobufjs lib.

Rather than using the standard Message.encode, you would call goEncode which will strip out any enum fields that are set to their default value (i.e. the value is 0).

function convertFields(obj, PB) {
  const converted = Object
    .keys(obj)
    .reduce((converted, field) => {
      const fieldType = PB.fields[field];

      if (fieldType) {
        const FieldPB = PB[fieldType.type];

        if (fieldType.resolvedType instanceof protobuf.Enum) {
          // If the field is an Enum and it's set to the first item (default item)
          // return the nothing so the field is not included in the resulting object.
          if (FieldPB && (obj[field] === 0)) {
            return converted;
          }
        } else if (fieldType.repeated) {
          converted[field] = obj[field]
            .map(fieldObj => (
              FieldPB ?
                convertFields(fieldObj, FieldPB) : fieldObj
            ));
        } else if (FieldPB) {
          converted[field] = convertFields(obj[field], FieldPB);
          return converted;
        }
      }

      converted[field] = obj[field];
      return converted;
    }, {});

  return converted;  
}

/*
 * Will encode a protobuf in a way that matches how GO does it.
 *
 * @param {object} message - A plain javascript object or protobuf instance.
 * @param {object} PB - The protobuf class that corresponds to the provided message.
 *
 * @returns {Uint8Array} - The encoded message.
 */
export function goEncode(message, PB) {
  let messageObj = message;

  if (message instanceof protobuf.Message) {
    messageObj = PB.toObject(message, {
      defaults: false,
      arrays: false,
      objects: false,
    });
  }

  const converted = convertFields(messageObj, PB);

  return PB.encode(converted).finish();
}
robmisio
  • 1,066
  • 2
  • 12
  • 20