Protobuf-net serialization/deserialization

Question

I checked but seem to be unable to see how to directly serialize a class to a byte array and subsequently deserialize from a byte array using Marc Gravell's protobuf-net implementation.

Edit: I changed the question and provided code because the original question of how to serialize into byte[] without having to go through stream was admittedly trivial. My apologies.

Updated Question: Is there any way to not have to deal with generics and instead infer the type of the property "MessageBody" through reflection when it is passed through the constructor? I assume I cannot serialize object type, correct? The current solution looks very cumbersome in that I need to pass in the type of the MessageBody each time I instantiate a new Message. Is there a sleeker solution to this?

I came up with the following:

class Program
{
    static void Main(string[] args)
    {
        Message<string> msg = new Message<string>("Producer", "Consumer", "Test Message");

        byte[] byteArray = msg.Serialize();
        Message<string> message = Message<string>.Deserialize(byteArray);

        Console.WriteLine("Output");
        Console.WriteLine(message.From);
        Console.WriteLine(message.To);
        Console.WriteLine(message.MessageBody);

        Console.ReadLine();

    }
}

[ProtoContract]
public class Message<T>
{
    [ProtoMember(1)]
    public string From { get; private set; }
    [ProtoMember(2)]
    public string To { get; private set; }
    [ProtoMember(3)]
    public T MessageBody { get; private set; }

    public Message()
    {

    }

    public Message(string from, string to, T messageBody)
    {
        this.From = from;
        this.To = to;
        this.MessageBody = messageBody;
    }

    public byte[] Serialize()
    {
        byte[] msgOut;

        using (var stream = new MemoryStream())
        {
            Serializer.Serialize(stream, this);
            msgOut = stream.GetBuffer();
        }

        return msgOut;
    }

    public static Message<T> Deserialize(byte[] message)
    {
        Message<T> msgOut;

        using (var stream = new MemoryStream(message))
        {
            msgOut = Serializer.Deserialize<Message<T>>(stream);
        }

        return msgOut;
    }   
}

What I like to get to is something such as:

Message newMsg = new Message("Producer", "Consumer", Foo); byte[] byteArray = newMsg.Serialize();

and Message msg = Message.Deserialize(byteArray);

(where Deserialize is a static method and it always deserializes into an object of type Message and only needs to know what type to deserialize the message body into).

Answer 1

there's a few different questions here, so I'll answer what I can see: if I've missed anything just let me know.

Firstly, as noted, a MemoryStream is the most common way of getting to a byte[]. This is consistent with most serializers - for example, XmlSerializer, BinaryFormatter and DataContractSerializer also don't have an "as a byte[] overload", but will accept MemoryStream.

Generics: you don't need to use generics; v1 has Serializer.NonGeneric, which wraps this away from you. In v2, the "core" is non-generic, and can be accessed via RuntimeTypeModel.Default; of course Serializer and Serializer.NonGeneric continue to work.

For the issue of having to include the type: yes, the protobuf spec assumes the receiver knows what type of data they are being given. A simple option here is to use a simple wrapper object as the "root" object, with multiple typed properties for the data (only one of which is non-null). Another option might spring from the inbuilt inheritance support via ProtoInclude (note: as an implementation detail, these two approaches are identical).

In your specific example, perhaps consider:

[ProtoContract]
[ProtoInclude(1, typeof(Message<Foo>))]
.... More as needed
[ProtoInclude(8, typeof(Message<Bar>))]
public abstract class Message
{   }
[ProtoContract]
public class Message<T> : Message
{
    ...
}

Then just serialize with <Message> - the API will create the right type automatically.

With recent builds, there is also a DynamicType option that includes type data for you, for example:

[ProtoContract]
public class MyRoot {
    [ProtoMember(1, DynamicType=true)]
    public object Value { get; set; }
}

This will work for any Value that holds a contract-type instance (but not for primitives, and ideally not involving inheritance).

Answer 2

The code the OP posted wouldn't quite work for me, the following is a slight adaptation taking onboard a little more of Marc Gravell's suggestions. Inheriting from Message was needed to prevent "Cyclic inheritance is not allowed", and as noted in the code comments below GetBuffer wasn't working out either.

Hoping it helps someone else, took me a good few hours to get it all to work...

      [ProtoContract]
      public abstract class Message
      {
        public byte[] Serialize()
        {
          byte[] result;
          using (var stream = new MemoryStream())
          {
            Serializer.Serialize(stream, this);
            result = stream.ToArray(); //GetBuffer was giving me a Protobuf.ProtoException of "Invalid field in source data: 0" when deserializing
          }
          return result;
        }
      }

      [ProtoContract]
      public class Message : Message
      {
        [ProtoMember(1)]
        public string From { get; private set; }
        [ProtoMember(2)]
        public string To { get; private set; }
        [ProtoMember(3)]
        public T MessageBody { get; private set; }

        public Message()
        { }

        public Message(string from, string to, T messageBody)
        {
          this.From = from;
          this.To = to;
          this.MessageBody = messageBody;
        }

        public static Message Deserialize(byte[] message)
        {
          Message result;
          using (var stream = new MemoryStream(message))
          {
            result = Serializer.Deserialize>(stream);
          }
          return result;
        }
      }

Answer 3

I also found that if using a .PROTO file (.cs compiled/generated with Google.Protobuf and Google.Protobuf.Tools packages), you'll use the record.WriteTo function in the serializer in and the static Parser.ParseFrom function in your Deserializer.

using Confluent.Kafka;
using System;
using System.IO;
using CodedOutputStream = Google.Protobuf.CodedOutputStream;
using SerializationContext = Confluent.Kafka.SerializationContext;

public class KafkaSerializer: ISerializer<Message>
{
    public byte[] Serialize(Message record, SerializationContext context)
    {
        using (var stream = new MemoryStream())
        {
            using (var codedStream = new CodedOutputStream(stream))
            {
                record.WriteTo(codedStream);
                codedStream.Flush();
                return stream.ToArray();
            }
        }
    }
}

public class KafkaDeserializer : IDeserializer<Message>
{
    public Message Deserialize(ReadOnlySpan<byte> data, bool isNull, SerializationContext context)
    {
        var msg = Message.Parser.ParseFrom(data);
        return msg;
    }
}