Protocol buffers detect type from raw message

Question

Is it possible to detect the type of a raw protocol buffer message (in byte[])

I have a situation where an endpoint can receive different messages and I need to be able to detect the type before I can deserialize it.

I am using protobuf-net

Answer 1

You can't detect the type in isolation, since the protobuf spec doesn't add any data to the stream for this; however, there are a number of ways of making this easy, depending on the context:

• a union type (as mentioned by Jon) covers a range of scenarios
• inheritance (protobuf-net specific) can be versatile - you can have a base-message type, and any number of concrete message types
• you can use a prefix to indicate the incoming type

the last approach is actually very valuable in the case of raw TCP streams; this is on the wire identical to the union type, but with a different implementation; by deciding in advance that 1=Foo, 2=Bar etc (exactly as you do for the union type approach), you can use SerializeWithLengthPrefix to write (specifying the 1/2/etc as the field number), and the non-generic TryDeserializeWithLengthPrefix to read (this is under Serializer.NonGeneric in the v1 API, or on the TypeModel in the v2 API), you can provide a type-map that resolves the numbers back to types, and hence deserialize the correct type. And to pre-empt the question "why is this useful with TCP streams?" - because: in an ongoing TCP stream you need to use the WithLengthPrefix methods anyway, to avoid over-reading the stream; so you might as well get the type identifier for free!

summary:

• union type: easy to implement; only down side is having to then check which of the properties is non-null
• inheritance: easy to implement; can use polymorphism or discriminator to handle "what now?"
• type prefix: a bit more fiddly to implement, but allows more flexibility, and has zero overhead on TCP streams

Answer 2

One typical option is to have a wrapper message to act as an "option type" or discriminated union. You could have an enum (one per message type) and a message containing a field with the message type in, and then one optional field per message type.

This is described in the Protobuf documentation as a "union type".

Answer 3

You could wrap it like this. Where data would hold the actual message.

message MyCustomProtocol {
  required int32 protocolVersion = 1;
  required int32 messageType = 2;
  bytes data = 3;
}

A general rule for protocols is to include a protocol version. You will be very happy to have it once you have old and new clients.

Answer 4

You could use a technique called Self Describing Messages. It can be used to generate a set of .proto files describing each message type encoded as 'any' within a wrapper. An example from the docs:

syntax = "proto3";

import "google/protobuf/any.proto";
import "google/protobuf/descriptor.proto";

message SelfDescribingMessage {
  // Set of FileDescriptorProtos which describe the type and its dependencies.
  google.protobuf.FileDescriptorSet descriptor_set = 1;

  // The message and its type, encoded as an Any message.
  google.protobuf.Any message = 2;
}

It should be noted that native support for these messages at the time of writing this response is only available in C++ and Java.