6

There's a neat option in protobuf-net ver 2 called [ProtoMember(2,AsReference=true)]. For the most part, this is a follow up question to:

I started wondering if reference integrity is always maintained irrespective of when serialization / deserialization occurs. Is protobuf-net already doing this when you use the AsReference option?

I threw together a basic code example for illustration purposes, and then thought, "perhaps I need to borrow some ideas from the ORM world?" Should I be implementing an identity map? Should I somehow tell protobuf (via a delegates?) to resolve references to/from foreign key values instead.

The answer I'd like to hear is that somehow protobuf-net can maintain reference integrity across assembly boundaries, even with types that just look like each other.

However, here's an alternative sequence just-in-case:

  • a => 1. resolve a reference to a primary key int, 2. serialize

  • b => 3. deserialize, 4. resolve primary key int to a reference

Notes/Constraints:

  • classes are a mirror image of each other, but are recompiled in each assembly.

  • the object graph needs to be the same irrespective of when the objects gets serialized. i.e. A.ref=B (serialize / deserialize). C.ref=B (serialize / deserialize).

Example for discussion:

using System;
using System.Collections.Generic;
using ProtoBuf;

namespace protobuf_question
{
    class Program
    {
        static void Main(string[] args)
        {
            var a = new A() { key = 1 };
            var b = new B() { A = a, key = 2 };
        }
    }

    [ProtoContract]
    public class A
    {
        [ProtoMember(1)]
        public int key { get; set; }
    }
    [ProtoContract]
    public class B
    {
        [ProtoMember(1)]
        public int key { get; set; }
        [ProtoMember(2,AsReference=true)]
        public A A { get; set; }  // a reference
    }
    [ProtoContract]
    public class IdentityMap<T,TKey>
    {        
        public static readonly IdentityMap<T,TKey> instance = new IdentityMap<T,TKey>();  // singleton
        private Dictionary<string, T> identitySpace { get; set; }
        public IEnumerable<string> GetIdentitySet (/* some criteria */)
        {            
            // used for serializing with reference safety between assemblies.
            throw new NotImplementedException();
        }
        public TKey GetKey(T reference)
        {
            // map object reference to identity map; return identity.
            throw new KeyNotFoundException();
        }
    }

}
Community
  • 1
  • 1
sgtz
  • 8,849
  • 9
  • 51
  • 91

1 Answers1

6

The answer I'd like to hear is that somehow protobuf-net can maintain reference integrity across assembly boundaries, even with types that just look like each other.

Yes, it can. When using AsReference=true it internally generates an opaque key unrelated to anything else, and uses that on the wire; this key is based on the deterministic nature of the stream. Everything remains 100% contract based. As long as the contracts are compatible it doesn't matter whether it is the same type, same process, same machine, same OS, etc.

The only exception to this is when using the DynamicType=true option, where it burns type metadata into the wire, but even that is not required to be the actual type data - there is an event you can subscribe to if you want to provide something more fine-grained and controlled than the default (type.AssemblyQualifiedName). If you were using DynamicType, this would allow you to transparently swap types between implementations. Of course, if you aren't using DynamicType then this problem doesn't exist in the first place.


Re your identity map... I'm unclear on the motivation there. But to be explicit: the existing code does not attempt to identify any "primary key" candidates and serialize just those; it serializes the object the first time it sees it (and invents a key), otherwise it just writes the key it made up last time.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • You sound solid on "the deterministic nature of the stream." It's going to take a little while for me to realise that I don't have to worry about this. Re: "Identity map". In the ORM world (such as nhibernate), a reference is mapped to an identity (or key). I'm not using nhibernate, but the concept seemed to be good way to extend protobuf-net beyond a perceived shortcomming... but it looks like I wont have to do this because you've thought of everything! Thanks again. – sgtz Jul 06 '11 at 13:01
  • say we want some degree of lazy loading on a reference. If we retrieve this referenced data via proto later, probably we will need to use that event you mentioned to relink it? Should we exclude such "lazy" references from a ProtoMember contract and handle manually? – sgtz Jul 06 '11 at 13:31
  • @sgtz the event is for the `DynamicType` case, not the reference-tracking; completely unrelated. protobuf doesn't have anything "lazy" build in, as it handles a single-pass unidirectional stream. You could, however, store the keys etc that you need as *regular* data on the type, and handle the lazy aspect yourself. Note that there is support for optional serialization, so you could choose to ***not*** serialize something that isn't loaded (and the receiver with have to load it), or *choose* to serialize it when you have it (and the receiver *won't* need to, as it will be in the stream). – Marc Gravell Jul 06 '11 at 13:55
  • I'm thinking that it would be useful to get access to the protobuf genereated key... even say "generate a key on X" even though X is not needed just yet. Is that possible? – sgtz Jul 06 '11 at 14:02
  • @sgtz not at current, and it would complicate the storage as I would then need a bucket per type (I.e. Customer 124 is not the same as Invoice 123). I'm not ruling it out, but no: that doesn't exist today. – Marc Gravell Jul 06 '11 at 15:09
  • btw: I opted to resolve this with an array of unique ids. At first I was going to use ObjectIDGenerator, but then decided to use the db foreign key value instead. The array is just a property right now. Potentially an annonymous type might offer for a cleaner implementation. – sgtz Jul 07 '11 at 17:22