Protobuf-net Error: Type is not expected, and no contract can be inferred: BlockHeader

Question

Trying to get de-serialization of an openstreetmap pbf file working properly by following information from this thread as well as other sources:

Protobuf-net Deserialize Open Street Maps

I'm currently using the protobug dll from r480. I used protogen to create the csharp class files from the osm proto's, but when I hit this point in the code where I'm trying to read in from the stream:

BlockHeader header;
using (var tmp = new LimitedStream(file, length))
{
    header = Serializer.Deserialize<BlockHeader>(tmp); // exception occurs here
}

It throws the following exception:

InnerException: System.InvalidOperationException
Message=Type is not expected, and no contract can be inferred: BlockHeader
Source=protobuf-net
StackTrace:
  at ProtoBuf.Meta.TypeModel.ThrowUnexpectedType(Type type) in C:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 1115
  at ProtoBuf.Meta.TypeModel.TryDeserializeAuxiliaryType(ProtoReader reader, DataFormat format, Int32 tag, Type type, Object& value, Boolean skipOtherFields, Boolean asListItem, Boolean autoCreate, Boolean insideList) in C:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 848
  at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in C:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 582
  at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in C:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 506
  at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type) in C:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 488
  at ProtoBuf.Serializer.Deserialize[T](Stream source) in C:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 69
  at OsmParserDemo.MainWindow.OsmParse() in C:\Users\crussell\Documents\Visual Studio 2010\Projects\OsmParseMadness\OsmParserDemo\MainWindow.xaml.cs:line 57
  at OsmParserDemo.MainWindow..ctor() in C:\Users\crussell\Documents\Visual Studio 2010\Projects\OsmParseMadness\OsmParserDemo\MainWindow.xaml.cs:line 28

So in comparing the code structure to that of the protobuf-net example I get the impression that I may be missing something in regards to identifying each member. Is this correct, or am I way out in space on this one? Any help or hints are greatly appreciated!

Edit: Here's a snippet of BlockHeader from the generated FileFormat.cs class:

BlockHeader

Hi. In your code, where has BlockHeader come from? IIRC it wasn't defined in the previous question. Has it come from protogen? Protoc? Or have you defined it manually? Or...? Any chance you can post what your BlockHeader looks like? — Marc Gravell, Jul 19 '12 at 18:43
Hi Marc! I copied the code that you and Jon wrote verbatim, and made a couple subtle changes to match v2 (such as removing the littleEndian conversion). BlockHeader wasn't described in detail in the previous post, however it refers to the BlockHeader class within Fileformat.cs which was generated using protogen v2.4.1.472. I will edit the initial post to include a portion of the class (it's quite large). — ninehundreds, Jul 19 '12 at 19:02
Your probably right. My understanding was in order to deserialize the osm pbf's you must first use protogen to generate the c# classes from the existing osm .proto files. Once you have the generated classes you use them as a definition when deserializing the osm.pbf. Was I supposed to use protobuf-net to generate the c# classes rather than protogen? I apologize if this is a daft question. — ninehundreds, Jul 19 '12 at 19:57
"protobuf-net rather than protogen" is a false question; there is more than one protogen; one of them is part of protobuf-net, and it would indeed be fine to use it. But another *is not* part of protobuf-net, and the output won't work with protobuf-net. More info in my answer. — Marc Gravell, Jul 19 '12 at 19:59
I had only grabbed the dll download, so that was a huge mistake. Now that I have downloaded your source repo I see the 'native' Protogen project. Once I generate these classes properly now I should have no issues. I appreciate your help very much, and will gladly buy you a pint or two if you are ever in NY. Cheers! — ninehundreds, Jul 19 '12 at 20:11
FWIW, you've inspired me to add some user guidance on this. There's a few new unit tests I've just added... let's just say: `[Test, ExpectedException(typeof(InvalidOperationException), ExpectedMessage = "Are you mixing protobuf-net and protobuf-csharp-port? See http://stackoverflow.com/q/11564914; type: Examples.Issues.SO11564914+BlockHeader")]` — Marc Gravell, Jul 19 '12 at 20:15
That would be really helpful. Although the examples on the website are easy to create and understand, I had trouble finding information specific to these osm files. I will do a brief tutorial on my website as well when I get it back online which will hopefully help some people. — ninehundreds, Jul 19 '12 at 20:21

score 17 · Accepted Answer · answered Jul 19 '12 at 19:54

With your addition of the code file, all becomes clear. The library you are using, based on the stack-trace, is protobuf-net; but that .cs file has nothing to do with protobuf-net. Well, very little.

You see, there are (at least) 2 c#/.net protobuf implementations, and I think you're mixing them up:

protobuf-csharp-port is by Jon Skeet, and is a direct port of the Java version, meaning: it follows the same design principles and API. Tempting if you work in both C# and Java.
protobuf-net is by me, and is a from-first-principles implementation that aims to be idiomatic .NET, meaning: it can work code-first, against your existing types (basically, like XmlSerializer, DataContractSerializer, etc can), but also supports usage from .proto if you want

Confusingly, both of them feature a code-generation tool called "protogen". AFAIK, this naming coincidence was just natural convergence/coincidence, rather than planning (good or malign).

A c# file generated from protobuf-csharp-port's protogen will work with the protobuf-csharp-port library

A c# file generated from protobuf-net's protogen will work with the protobuf-net library

Here's the protobuf-net generated version, including BlockHeader:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

// Generated from: Foo.proto
namespace ConsoleApplication9
{
  [global::System.Serializable, global::ProtoBuf.ProtoContract(Name=@"Blob")]
  public partial class Blob : global::ProtoBuf.IExtensible
  {
    public Blob() {}


    private byte[] _raw = null;
    [global::ProtoBuf.ProtoMember(1, IsRequired = false, Name=@"raw", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue(null)]
    public byte[] raw
    {
      get { return _raw; }
      set { _raw = value; }
    }

    private int _raw_size = default(int);
    [global::ProtoBuf.ProtoMember(2, IsRequired = false, Name=@"raw_size", DataFormat = global::ProtoBuf.DataFormat.TwosComplement)]
    [global::System.ComponentModel.DefaultValue(default(int))]
    public int raw_size
    {
      get { return _raw_size; }
      set { _raw_size = value; }
    }

    private byte[] _zlib_data = null;
    [global::ProtoBuf.ProtoMember(3, IsRequired = false, Name=@"zlib_data", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue(null)]
    public byte[] zlib_data
    {
      get { return _zlib_data; }
      set { _zlib_data = value; }
    }

    private byte[] _lzma_data = null;
    [global::ProtoBuf.ProtoMember(4, IsRequired = false, Name=@"lzma_data", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue(null)]
    public byte[] lzma_data
    {
      get { return _lzma_data; }
      set { _lzma_data = value; }
    }

    private byte[] _bzip2_data = null;
    [global::ProtoBuf.ProtoMember(5, IsRequired = false, Name=@"bzip2_data", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue(null)]
    public byte[] bzip2_data
    {
      get { return _bzip2_data; }
      set { _bzip2_data = value; }
    }
    private global::ProtoBuf.IExtension extensionObject;
    global::ProtoBuf.IExtension global::ProtoBuf.IExtensible.GetExtensionObject(bool createIfMissing)
      { return global::ProtoBuf.Extensible.GetExtensionObject(ref extensionObject, createIfMissing); }
  }

  [global::System.Serializable, global::ProtoBuf.ProtoContract(Name=@"BlockHeader")]
  public partial class BlockHeader : global::ProtoBuf.IExtensible
  {
    public BlockHeader() {}

    private string _type;
    [global::ProtoBuf.ProtoMember(1, IsRequired = true, Name=@"type", DataFormat = global::ProtoBuf.DataFormat.Default)]
    public string type
    {
      get { return _type; }
      set { _type = value; }
    }

    private byte[] _indexdata = null;
    [global::ProtoBuf.ProtoMember(2, IsRequired = false, Name=@"indexdata", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue(null)]
    public byte[] indexdata
    {
      get { return _indexdata; }
      set { _indexdata = value; }
    }
    private int _datasize;
    [global::ProtoBuf.ProtoMember(3, IsRequired = true, Name=@"datasize", DataFormat = global::ProtoBuf.DataFormat.TwosComplement)]
    public int datasize
    {
      get { return _datasize; }
      set { _datasize = value; }
    }
    private global::ProtoBuf.IExtension extensionObject;
    global::ProtoBuf.IExtension global::ProtoBuf.IExtensible.GetExtensionObject(bool createIfMissing)
      { return global::ProtoBuf.Extensible.GetExtensionObject(ref extensionObject, createIfMissing); }
  }

}

Conclusion:

Either use the protobuf-net protogen, or use the protobuf-csharp-port library. No mix and match.

As a side note rather than building your Protogen from source I located the executable in your r282.zip within the NET30 archive (worked great too BTW!). It would be helpful to have it added into the next available release download alongside the protobuf-net.dll for those of us who don't necessarily want or need to compile the library. — ninehundreds, Jul 19 '12 at 21:56

kravasb · Answer 2 · 2013-07-10T11:04:46.983

I had this kind of error when running this code:

public IEnumerable<T> Deserialize<T>(string path)
{
    using (var stream = File.Open(path, FileMode.Open, FileAccess.Read))
    {
        var item = Serializer.Deserialize<IEnumerable<T>>(stream);
        return item;
    }
}

When I changed IEnumerable<T> to List<T> the problem dissappeared. I guess that ProtoBuff knows nothing about interfaces and you should specify non abstract/interface type you want deserialize to.

UPD: To conclude: You should deserialize into the type object was serialized. No base types.

Protobuf-net Error: Type is not expected, and no contract can be inferred: BlockHeader

2 Answers2