GRPC: Passing JSON with proto.Message field between browser and server

Question

I've been struggling with passing a JSON with Message values back and forth between systems. Got a bit further, but I'm still not there. A Struct seems to be the way but even though the struct I'm sending looks perfectly fine, it is empty once received by the server.

Result is passed from a web browser (grpc-web) to a Python backend. The Python backend should serialize Result to JSON (to store it) and back again.

// proto
message Result {
    google.protobuf.Struct variables = 1;
}

// obj - Where variables would contain a (1 level deep) JSON with different types of values, e.g.:
{
    "key1": 1,
    "key2": true,
    "key3": proto_msg_a //instance of proto.MessageA
}

// code
struct = new proto.google.protobuf.Struct(obj);
req = new Request;
req.variables = struct;

Checking req.variables before sending shows that it's indeed a Struct with all the correct fields in it. But once the other end (server) receives it req.variables is an empty Struct. For testing purposes I tried an obj that is simply {'key': 'value'}, but the result was the same.

So then I tried proto.google.protobuf.Struct.fromJavaScript:

// code
struct = proto.google.protobuf.Struct.fromJavaScript(vars);
req = new Request;
req.variables = struct;

This works for a simple obj (e.g. {"key": "val"}), but for an obj with a proto message field (such as above) it resulted in :

struct_pb.js:875 Uncaught Error: Unexpected struct type.
    at Function.proto.google.protobuf.Value.fromJavaScript (struct_pb.js:875)
    at Function.proto.google.protobuf.Struct.fromJavaScript (struct_pb.js:941)
    at Function.proto.google.protobuf.Value.fromJavaScript (struct_pb.js:871)
    at Function.proto.google.protobuf.Struct.fromJavaScript (struct_pb.js:941)
    at Function.proto.google.protobuf.Value.fromJavaScript (struct_pb.js:871)
    at Function.proto.google.protobuf.Struct.fromJavaScript (struct_pb.js:941)
    at Function.proto.google.protobuf.Value.fromJavaScript (struct_pb.js:871)
    at Function.proto.google.protobuf.Struct.fromJavaScript (struct_pb.js:941)

Or can I, instead of going through all the troubles with protobuf/json in javascript, just use a map?

// proto
message Request {
    map<string, ?type?> variables = 1;
}

But what would ?type? then be if it the values can ben anything (proto.MessageX, string, boolean, etc)?

I would really like to use proto messages in variables. The reason for picking protobuf/grpc was exactly this, being able to use the same type throughout our complete platform, but this seems to be blocking this goal. Did I miss something? What would you do?

score 2 · Answer 1 · answered Jun 15 '21 at 17:24

I'm not answering your question exactly, but I was having an issue with Struct.fromJavaScript that might be somewhat related.

TL;DR

You can't have undefined (or any invalid JSON) in whatever you pass to Struct.fromJavaScript. This includes any deeply nested fields if you're passing an object.

The way I'm fixing this for my solution is to JSON.stringify and JSON.parse the object I want to turn into a Struct before passing it to Struct.fromJavaScript to remove all fields with invalid JSON:

const obj = { ... }

const stringifiedJSON = JSON.stringify(obj)
const parsedJSON = JSON.parse(stringifiedJSON)

const newStruct = Struct.fromJavaScript(parsedJSON)

I was looking around for what causes this error for a while, and it looks like in the struct implementation of google-protobuf, there's a switch statement that looks something like this:

proto.google.protobuf.Value.fromJavaScript = function(value) {
  var ret = new proto.google.protobuf.Value();
  switch (goog.typeOf(value)) {
    case 'string':
      ret.setStringValue(/** @type {string} */ (value));
      break;
    case 'number':
      ret.setNumberValue(/** @type {number} */ (value));
      break;
    case 'boolean':
      ret.setBoolValue(/** @type {boolean} */ (value));
      break;
    case 'null':
      ret.setNullValue(proto.google.protobuf.NullValue.NULL_VALUE);
      break;
    case 'array':
      ret.setListValue(proto.google.protobuf.ListValue.fromJavaScript(
          /** @type{!Array} */ (value)));
      break;
    case 'object':
      ret.setStructValue(proto.google.protobuf.Struct.fromJavaScript(
          /** @type{!Object} */ (value)));
      break;
    default:
      throw new Error('Unexpected struct type.');
  }
  return ret;
};

It looks like it just goes through the object fields and checks the types of the fields, then it will throw the Unexpected struct type error if the type is not one of the previous cases. This means that if the field is undefined, the error will throw.

Also, the type definition on Struct.fromJavaScript is this:

(method) Struct.fromJavaScript(value: {
    [key: string]: JavaScriptValue;
}): Struct

where JavaScriptValue is

export type JavaScriptValue = null | number | string | boolean | Array<any> | {};

And if you pass an object with undefined into fromJavaScript directly instead of through a typed variable...

Struct.fromJavaScript({
  field: undefined, // this will cause a compile-time error
})

// ----------

type WeirdObj = {
  field?: string
}

const obj: WeirdObj = {
  field: undefined,
}

Struct.fromJavaScript(obj) // this will throw a run-time error

// ----------

type WeirdUndefinedObj = {
  field: string | undefined
}

const undefinedObj: WeirdUndefinedObj = {
  field: undefined,
}

Struct.fromJavaScript(undefinedObj) // this will cause a compile-time error

This confused me for a while, because fromJavaScript (to me) seems like I can pass it JavaScript, and I'll get a struct back in return. Maybe the method should be fromJSON instead, and update the types to prevent undefined from getting passed into the method in the first place.

GRPC: Passing JSON with proto.Message field between browser and server

1 Answers1