Protobuf-net: Serializing a 3rd party class with a Stream data member

Question

How do you serialize a Stream (or more correctly Stream derived) data member of a class?

Assuming we have a 3rd Party class that we can't attribute:

public class Fubar
{
    public Fubar() { ... }
    public string Label { get; set; }
    public int DataType { get; set; }
    public Stream Data { get; set; } // Where it's always actually MemoryStream
};

I'm trying to use protobuf-net to serialize the class. Working through the exceptions and various SO questions I've come up with:

RuntimeTypeModel.Default.Add(typeof(Stream), true)
    .AddSubType(1, typeof(MemoryStream));
RuntimeTypeModel.Default.Add(typeof(Fubar), false)
    .Add(1, "Label")
    .Add(2, "DataType")
    .Add(3, "Data");

using (MemoryStream ms = new MemoryStream())
{
    Fubar f1 = new Fubar();
    /* f1 initialized */

    // Serialize f1
    Serializer.SerializeWithLengthPrefix<Message>(ms, f1, PrefixStyle.Base128);

    // Now let's de-serialize
    ms.Position = 0;
    Fubar f2 = Serializer.DeserializeWithLengthPrefix<Fubar>(ms, PrefixStyle.Base128);
}

The above runs with no errors. The Label and DataType are correct in f2 but the Data variable is just an empty stream. Debugging the code I see that the memory stream is something like 29 bytes (while the Data stream in f1 itself is over 77KiB).

I feel as if I'm missing something fairly trivial but just can't seem to figure out what it would be. I assume that it is indeed possible to serialize a stream data member. Do I have to perhaps somehow specify the data properties for the Stream or MemoryStream types as well?

score 3 · Accepted Answer · answered Jun 26 '13 at 20:53

Stream is a very complex beast, and there is no inbuilt serialization mechanism for that. Your code configures it as though it were a type with no interesting members, which is why it is coming back as empty.

For this scenario, I'd probably create a surrogate, and set it up with just:

RuntimeTypeModel.Default.Add(typeof(Fubar), false)
       .SetSurrogate(typeof(FubarSurrogate));

where:

[ProtoContract]
public class FubarSurrogate
{
    [ProtoMember(1)]
    public string Label { get; set; }
    [ProtoMember(2)]
    public int DataType { get; set; }
    [ProtoMember(3)]
    public byte[] Data { get; set; }

    public static explicit operator Fubar(FubarSurrogate value)
    {
        if(value == null) return null;
        return new Fubar {
            Label = value.Label,
            DataType = value.DataType,
            Data = value.Data == null ? null : new MemoryStream(value.Data)
        };
    }
    public static explicit operator FubarSurrogate(Fubar value)
    {
        if (value == null) return null;
        return new FubarSurrogate
        {
            Label = value.Label,
            DataType = value.DataType,
            Data = value.Data == null ?
                 null : ((MemoryStream)value.Data).ToArray()
        };
    }
}

Thanks for the quick response. Well that certainly is the "something simple" I was missing. I was hoping not to have to go the surrogate route as the actual class involved has 20-30 properties some of which may or may not be set/used depending on the configuration/filter. Is there no way to specify some sort of surrogate for the Stream or MemoryStream class to help the serialization/deserialization? — CWoods, Jun 26 '13 at 21:48
@CWoods ... Well, maybe you could surrogate Stream? But that suggestion makes me want to claw my hands off... — Marc Gravell, Jun 26 '13 at 21:50
Wouldn't want you to lose the hands... Just didn't know if there was a way to get a helper into the mix based on the class type allowing me to avoid the surrogate route for the whole data class. I'll just bite the bullet and create a surrogate for the whole thing. — CWoods, Jun 26 '13 at 21:53

score 1 · Answer 2 · answered Jun 27 '13 at 03:47

Not to make Marc claw his own hands off... but in case anyone else wants to create a surrogate for Stream I've adapted Marc's surrogate example from the answer:

[ProtoContract]
public class StreamSurrogate
{
    [ProtoMember(1)]
    public byte[] Data { get; set; }

    public static explicit operator Stream(StreamSurrogate value)
    {
        if (value == null)
        {
            return null;
        }

        return new MemoryStream(value.Data);
}

    public static explicit operator StreamSurrogate(Stream value)
    {
        if (value == null)
        {
            return null;
        }

        if (value is MemoryStream)
        {
            return new StreamSurrogate { Data = ((MemoryStream)value).ToArray() };
        }
        else
        {
            // Probably a better way to do this...
            StreamSurrogate ss = new StreamSurrogate();

            ss.Data = new byte[value.Length];
            value.Read(ss.Data, 0, (int)value.Length);
            return ss;
        }
    }
}

And then for the RuntimeTypeModel:

MetaType mt2 = RuntimeTypeModel.Default.Add(typeof(Stream), true);

    mt2.AddSubType(1, typeof(MemoryStream));
    mt2.SetSurrogate(typeof(StreamSurrogate));

In which case f2 in my example appears to be fully correct!

Yes there are a lot of potential troubles with trying to serialize Stream - perhaps it would be wiser for me to set the surrogate on the MemoryStream subtype only since I know my particular case will always be using MemoryStreams for Data. However my line of thinking for registering the surrogate on Stream is as follows:

The data member of the class is a Stream.
Any Stream derived class could have been used on the original object and most of them likely can't be recreated.
However since the data member is a Stream, any Stream derived class should suffice for the deserialized object (since it only must support Stream).
MemoryStream is probably the best candidate in most cases for the deserialized stream.

Protobuf-net: Serializing a 3rd party class with a Stream data member

2 Answers2

Linked