Protobuf-net deserializes empty collection to null when the collection is a property of another type

Question

I've run in to a problem with protobuf-net and I hope it is user error and not a bug with protobuf-net.

I can serialize an empty collection (for example, IDictionary<string, string>()) and then deserialize the collection. The deserialization results in a non-null object (exactly as I serialized).

However, things get wonky if the collection belongs to another type. Serializing the custom type with a non-null empty collection results in a deserialization where the collection is null.

Below is an example of what I am working on and unit tests that illustrate the problem.

The test Test_Protobuf_EmptyDictionary_SerializeDeserialize passes. The test Test_Protobuf_EmptyCollectionAndPopulatedCollection_SerializeDeserialize fails.

[ProtoContract]
[ProtoInclude(100, typeof(ConcreteA))]
public abstract class AbstractClass
{
    [ProtoMember(1)]
    public string Name { get; set; }

    [ProtoMember(2)]
    public IDictionary<string, string> FieldsA { get; set; }

    [ProtoMember(3)]
    public IDictionary<string, string> FieldsB { get; set; }

    [ProtoMember(4)]
    public ICollection<string> FieldsC { get; set; }

    [ProtoMember(5)]
    public ICollection<string> FieldsD { get; set; } 
}

[ProtoContract]
[ProtoInclude(110, typeof(ConcreteB))]
public class ConcreteA : AbstractClass
{
    public ConcreteA() {}
}

[ProtoContract]
[ProtoInclude(120, typeof(ConcreteC))]
public class ConcreteB : ConcreteA
{
    [ProtoMember(1)]
    public int Age { get; set; }

    public ConcreteB() {}
}

[ProtoContract]
public class ConcreteC : ConcreteB
{
    [ProtoMember(1)]
    public string HairColor { get; set; }
}

[TestFixture]
public class ProtobufTests
{
    [Test]
    public void Test_Protobuf_EmptyDictionary_SerializeDeserialize()
    {
        IDictionary<string,string> dictionary = new Dictionary<string, string>();
        ICollection<string> collection = new List<string>();

        Assert.IsNotNull(dictionary);
        Assert.IsNotNull(collection);
        Assert.AreEqual(0, dictionary.Keys.Count);
        Assert.AreEqual(0, collection.Count);

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, dictionary);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<IDictionary<string, string>>(ms);

            Assert.IsNotNull(deserialized);
            Assert.AreEqual(0, deserialized.Keys.Count);
        }

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, collection);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<ICollection<string>>(ms);

            Assert.IsNotNull(deserialized);
            Assert.AreEqual(0, deserialized.Count);
        }
    }

    [Test]
    public void Test_Protobuf_EmptyCollectionAndPopulatedCollection_SerializeDeserialize()
    {
        ConcreteC c = new ConcreteC {
                                        FieldsA = new Dictionary<string, string>(),
                                        FieldsB = new Dictionary<string, string> {{"john", "elway"}},
                                        FieldsC = new List<string>(),
                                        FieldsD = new List<string>{"james", "jones"}
                                    };
        Assert.IsNotNull(c);
        Assert.IsNotNull(c.FieldsA);
        Assert.IsNotNull(c.FieldsB);
        Assert.IsNotNull(c.FieldsC);
        Assert.IsNotNull(c.FieldsD);
        Assert.AreEqual(0, c.FieldsA.Keys.Count);
        Assert.AreEqual(1, c.FieldsB.Keys.Count);
        Assert.AreEqual(0, c.FieldsC.Count);
        Assert.AreEqual(2, c.FieldsD.Count);

        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, c);
            ms.Position = 0;
            var deserialized = Serializer.Deserialize<ConcreteC>(ms);

            Assert.IsNotNull(deserialized);
            Assert.IsNotNull(deserialized.FieldsA); // first failing test
            Assert.IsNotNull(deserialized.FieldsB);
            Assert.IsNotNull(deserialized.FieldsC);
            Assert.IsNotNull(deserialized.FieldsD);
            Assert.AreEqual(0, deserialized.FieldsA.Keys.Count);
            Assert.AreEqual(1, deserialized.FieldsB.Keys.Count);
            Assert.AreEqual(0, deserialized.FieldsC.Count);
            Assert.AreEqual(1, deserialized.FieldsD.Count);
        }
    }
}

score 21 · Accepted Answer · answered Feb 07 '14 at 15:52

Ultimately, this comes down to the fact that the protobuf specification has no concept of null, and no way of expressing null. A collection is really just a repeated block (from the .proto specification); and a repeated block of 0 items is nothing whatsoever; no bytes; nothing. Since the collection member itself does not appear in protobuf, there is nowhere to say whether that thing is null vs not-null.

In xml terms, if is like using [XmlElement] to embed child items inside a parent (rather than [XmlArray] / [XmlArrayItem] - i.e.

<Foo>
    <Name>abc</Name>
    <Item>x</Item>
    <Item>y</Item>
    <Item>z</Item>
</Foo>

Here, a Foo with 0 Items would be:

<Foo>
    <Name>abc</Name>
</Foo>

from which, it is impossible to say whether the collection itself was null vs empty. Obviously protobuf isn't actually xml, but the above is meant purely as an illustrative example.

So: protobuf has no way to express this scenario, and therefore neither does protobuf-net.

In the other scenario you represent: the serialized representation of an empty list (as the root element) is: zero bytes. When you deserialize, if it finds the stream is empty, it always returns a non-null value as the root object, that being the most likely version of what you wanted.

Is there a way to change this behavior? Types that go over-the-wire which is a big use case of Protobuf tend to be immutable and usually refrain fron using Null. I would argue null should never be used and having an empty list instead should be the default value; at the very least for collection IEnumerable based fields on a type an empty list for collections would be the more common case for users. Is there an attribute or something that we could apply to the field to distinguish this case on a field-by-field basis? — akara, Sep 15 '16 at 04:28
@akara not currently; however, you can just initialize the property in the constructor - or in C# "current": `List Foos {get;} = new List();` — Marc Gravell, Sep 15 '16 at 10:52
@MarcGravell Nice explanation with XmlArrayAttribute. Only difference I can think of in terms of behavior; XML Deserialize will initialize all the collection to empty collection instead of null. — Riju, Jun 24 '20 at 02:12

score 1 · Answer 2 · answered Mar 20 '20 at 05:41

1

make sure you have a public default constructor so protobuf-net can give you an empty object.

answered Mar 20 '20 at 05:41

increddibelly

1,221
1
15
25

Protobuf-net deserializes empty collection to null when the collection is a property of another type

2 Answers2