protobuf-net inheritance

Question

Marc mentioned on stackoverflow that it will be possible in v2 of protobuf-net to use ProtoInclude attribute (or similar approach) to serialize/deserialize class hierarchy without a need to specify each subtype in the base class. Is this implemented yet? We have a plugin interface that can be derived in external libraries, so there is no way of knowing what the derived types will be. We could maintain unique numbering between types though, but I couldn’t find any examples on the net, short of using ProtoInclude attribute which requires a subtype to be specified.

How would I go about implementing inheritance with protobuf-net like that if I don't know what the subtypes are?

Answer 1

If you can't specify the subtypes in attributes (because it isn't known at compile-time) you have 2 options (both of which only apply to "v2", available as beta):

1. use a RuntimeTypeModel, rather than the static Serializer methods (which are now just a short-cut to RuntimeTypeModel.Default); tell the model about the inheritance (example below)
2. add DynamicType = true to the [ProtoMember(...)] in question

The second is not very pure protobuf - it embeds type information, which I don't really love but people just kept asking for. The first is my preferred option. To add subtypes at runtime:

var model = TypeModel.Create();
var type = model.Add(typeof(YourBaseType), true);
var subTypeA = model.Add(typeof(SomeSubType), true);
var subTypeB = model.Add(typeof(SomeOtherSubType), true);
type.AddSubType(4, typeof(SomeSubType));
type.AddSubType(5, typeof(SomeOtherSubType));

the true in the above means "use normal rules to add member properties automatically" - you can also take control of that and specify the properties (etc) manually if you prefer.

Note that a TypeModel should be cached and re-used (not created per object you need to serialize), as it includes some "emit" code to generate methods. Re-using it will be faster and require less memory. The type-model is thread-safe, and can be used to serialize/deserialize multiple streams concurrently on different threads.

Answer 2

To further expand Marc's answer, specifically dealing with RuntimeTypeModel, this is one way to write it:

RuntimeTypeModel.Default[typeof(BaseClass)].AddSubType(20, typeof(DerivedClass));

If you have more classes derived from derived classes, link them like this

RuntimeTypeModel.Default[typeof(DerivedClass)].AddSubType(20, typeof(DerivedFromDerivedClass ));

And so on.
You can then use Serializer.Serialize(file,object), as you would normally with protobuf-net.
This works across projects and namespaces.

Answer 3

By adding helper extension method:

public static class RuntimeTypeModelExt
{
    public static MetaType Add<T>(this RuntimeTypeModel model)
    {
        var publicFields = typeof(T).GetFields().Select(x => x.Name).ToArray();
        return model.Add(typeof(T), false).Add(publicFields);
    }
}

You can streamline sub-type registration like this:

private static RuntimeTypeModel CreateModel()
{
    var model = TypeModel.Create();
    model.Add<ExportInfo>();
    model.Add<RegistrationInfo>();
    model.Add<FactorySetupInfo>()
        .AddSubType(101, model.Add<ServiceSetupInfo>().Type)
        .AddSubType(102, model.Add<GenericWrapperSetupInfo>().Type)
        .AddSubType(103, model.Add<DecoratorSetupInfo>().Type);
    return model;
}

Answer 4

I solved this by completely flattening the hierarchy of fields for each ProtoContract. I also assume all fields will be serialized unless [ProtoIgnore] exists and therefore do not decorate fields with [ProtoMember(n)]. It does not need oneOf or [ProtoInclude]. This is what I use, no more scratching my head wondering what went wrong in the numbering etc. Feel free to expand on it to your needs.

A note: it takes any Type decorated with ProtoContract in an Assembly, flattens it's fields, ignores those with [ProtoIgnore], then automatically builds up all the ProtoMembers(n) (so no need for manual numbering).

It also accepts a single Generic Argument Protocol, where you supply the known Generic types e.g MyContract'1 and you supply typeof(MyContract2) such that the concreted type for serializing is MyContract<MyContract2>

Edit: This is a basic example and serves my needs. You may wish to extend it and sort the fields like so, so that you can manage adding new fields and retain backwards compat.

https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/data-member-order

To this end you could ignore Proto attributes entirely in the code and replace with DataContract, DataMember etc for discovery and let your contracts be both DataContracts and ProtoContracts with known ordering for both.

    /// <summary>
    /// <see cref="CompileProtocols(Assembly, Type[])"/>.
    /// </summary>
    /// <param name="knownGenericArguments"></param>
    /// <returns></returns>
    public static IEnumerable<Type> CompileProtocols(params Type[] knownGenericArguments)
    {
        var rv = CompileProtocols(Assembly.GetExecutingAssembly(), knownGenericArguments);
        return rv;
    }

    /// <summary>
    /// Compiles protocols that are non generic or take a single generic argument (`1). Single generic argument protocols
    /// will be concreted against the knownGenericArguments passed into the method.
    /// </summary>
    /// <param name="genericArgumentTypes"></param>
    public static IEnumerable<Type> CompileProtocols(Assembly inAssembly, params Type[] knownGenericArguments)
    {
        var runtimeProtocolTypes = new List<Type>();

        var assemblyProtocolTypes =inAssembly.GetTypes().Where(t => t.GetCustomAttribute<ProtoContractAttribute>() != null);            

        foreach(var assemblyProtocolType in assemblyProtocolTypes)
        {
            if(assemblyProtocolType.IsGenericType == false)
            {
                runtimeProtocolTypes.Add(assemblyProtocolType);
            }
            else
            {
                if (knownGenericArguments.Length > 0)
                {
                    var assemblyTypeGenericArgs = assemblyProtocolType.GetGenericArguments();

                    if (assemblyTypeGenericArgs.Length == 1)
                    {
                        foreach (var knownGenericArgument in knownGenericArguments)
                        {
                            var runtimeGenericType = assemblyProtocolType.MakeGenericType(knownGenericArgument);
                            runtimeProtocolTypes.Add(runtimeGenericType);
                        }
                    }
                }
            }
        }

        BuildModel(runtimeProtocolTypes);

        return runtimeProtocolTypes;
    }

    /// <summary>
    /// Builds and optionally compiles (default=true) the RuntimeTypeModel. In this case it uses the RumtimeTypeModel.Default
    /// with a CompileInPlace.
    /// 
    /// Note: You might wish to change this to build another model or return a complete Compiled model instead.
    /// </summary>
    /// <param name="protocolTypes"></param>
    /// <param name="compile"></param>
    private static void BuildModel(IEnumerable<Type> protocolTypes, bool compile = true)
    {
        foreach (var protocolType in protocolTypes)
        {
            int index = 1;

            var fields = protocolType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.FlattenHierarchy).Where(f => f.GetCustomAttribute<ProtoIgnoreAttribute>() == null);                    
            var metaType = RuntimeTypeModel.Default.Add(protocolType);

            foreach (var field in fields)
            {
                metaType.AddField(index, field.Name);
                index++;
            }
        }

        if (compile)
        {
            RuntimeTypeModel.Default.CompileInPlace();
        }
    }