8

I am sometimes receiving the following exception when attempting to deserialise an object using protobuf.net. I'm surprised as I never have more than a single thread deserialising the same object at the same time and the protobuf.net source does not seem to use any static objects for deserialising. The exception does suggest a solution but I am unsure as to how to implement so would welcome an example.

Base Exception Type:
System.TimeoutException: Timeout while inspecting metadata; this may indicate a deadlock. This can often be avoided by preparing necessary serializers during application initialization, rather than allowing multiple threads to perform the initial metadata inspection
at ProtoBuf.Meta.RuntimeTypeModel.TakeLock(Boolean& lockTaken)
at ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto(Type type, Boolean demand, Boolean addWithContractOnly, Boolean addEvenIfAutoDisabled)
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)

Inner Exception Type:
System.TimeoutException: Timeout while inspecting metadata; this may indicate a deadlock. This can often be avoided by preparing necessary serializers during application initialization, rather than allowing multiple threads to perform the initial metadata inspection
at ProtoBuf.Meta.RuntimeTypeModel.TakeLock(Boolean& lockTaken)
at ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto(Type type, Boolean demand, Boolean addWithContractOnly, Boolean addEvenIfAutoDisabled)
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)

Stack Trace: 
at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)
at ProtoBuf.Meta.TypeModel.GetKey(Type& type)
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type)

Regards, Marc

edit to add: I define my serialisable objects thus:

[ProtoContract]
public class Job
{
    [ProtoMember(1)]
    public long JobId { get; private set; } 
}

It would be difficult for me to easily call PrepareSerialiser on each one of my serialisable objects as I have many in different namespaces. However thinking about it what happens if protobuf is asked to deserialise two objects of the same type, a type it has not seen before, at exactly the same time?

MarcF
  • 3,169
  • 2
  • 30
  • 57
  • Heya Marc. If I could reliably reproduce the problem I would certainly provide more information. Unfortunately this happens in such a small percentage (0.01%) of cases it's proving **very** difficult to pin down. All of the objects I serialise are relatively small perhaps 10 value types and a couple of lists. I will find out where I can add the prepareSerializer call and let you know if the problem still occurs. – MarcF Sep 10 '11 at 15:45
  • I am deeply intrigued... I will try to browse the code for anything that could cause this (thanks for including the full stack trace, btw - that helps). This is.... odd. – Marc Gravell Sep 10 '11 at 16:05
  • It is worth adding that while I only ever have one thread deserialising a unique byte[], I may have up to 100 threads deserialising 100 unique byte[] at the same moment. Could it be an issure related to CPU load, i.e. the time taken between start deserialise and end being too long? – MarcF Sep 10 '11 at 16:09

4 Answers4

9

Old question, but if somebody happens to get this error, please check the DLL version you are using. Chances for this exception to show up in portable version are very high.

There are a couple of PRs related to this issue with the Portable version that are https://github.com/mgravell/protobuf-net/pull/98 and https://github.com/mgravell/protobuf-net/pull/114.

user2480322
  • 3
  • 2
Krzysztof Cygan
  • 310
  • 3
  • 9
  • 3
    I've experienced this pretty consistently with 2.0.0.640 PCL library in a threaded environment. Adding references to explicit versions in higher level projects solves this by replacing the PCL version with a version specific to that higher level assembly (SL, .Net 40, etc) which immediately solves the issue. – Tarwn Sep 03 '13 at 20:21
  • This answer and comment saved me a bunch of time. Great insights, friends. – Arielr Sep 16 '13 at 10:10
  • This error consistently occurs for me with PCL version 2.0.0.668. I have also tried bumping up the timeout @MarcGravell suggested above which does not solve it. Any clues other than switching to respective CLR specific version of the .DLL? – siva Jun 05 '15 at 05:15
  • In 2016 that is still the issue. I accidentally replaced normal 2.0.0.668 with PCL version on one deployment and immediatly many of those timeout errors appeared. Removing PCL helped to solve the issue. – Evk Jun 02 '16 at 12:43
2

For me preparing Serializers using

Serializer.PrepareSerializer<Type>();

didn't really helped me.

For me, solution (aka workaround) was to serialize type at startup which was causing trouble at app startup :

MessageSerialization.Serialize(new Type());

Where MessageSerialization.Serialize is serialization method using protobuf Serializer.Serialize(stream, o)

Michal Dobrodenka
  • 1,104
  • 8
  • 27
2

RuntimeTypeModel.Default (the default model) is static, and underpins the static Serializer class (re the comment that there wasn't any static state). Despite adding this check due to paranoia, I have never seen this error raised. I would absolutely love to see an example that will repro this. Are you sure you aren't threading? If not for threading, I can only wonder: is the type-model really really big?

Indeed, even with many threads hitting it aggressively at startup (i.e. here on stackoverflow) it is well behaved. As the error message hints, you might try calling Serializer.PrepareSerializer during app-startup, which will pre-initialize everything, avoiding any threading concerns.

But hey! At least it didn't deadlock!

The odd thing here, though, is that it still shouldn't be possible to deadlock - it deliberately uses a coarse lock to avoid issues from the order it takes lock. Again - I'd really love to see a sample.

musicakc
  • 488
  • 1
  • 4
  • 15
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • 1
    @MarcF - as a random thing, can you try setting `RuntimeTypeModel.Default.MetadataTimeoutMilliseconds` to something stupidly big, just to see if this is a genuine deadlock vs something taking longer than normal? – Marc Gravell Sep 16 '11 at 12:03
  • Sorry this reply took so long. I've just gone over some of my old stackoverflow posts. I did indeed increase the MetadataTimeoutMilliseconds to '150000' and I don't think I have seen the error since. – MarcF Dec 12 '11 at 14:57
  • just thought i'd add i've just hit this error too.. Only 1 time in thousands of test, but I do have multiple threads doing the work. (4 atm, but will grow as more things come to life) – Paul Farry Mar 23 '12 at 01:57
  • This just happened to me when performing CPU bound work on all cores. My entire machine was slow. Many threads were trying to take the lock and one of them was in some "compile" method. – usr Apr 30 '13 at 17:42
  • @usr if it is a false positive caused by very high (but unrelated) CPU, then increasing the timeout is the simplest option. Alternatively, if this is a likely scenario, you might consider the precompiler (included in the google-code download), which can create a pre-baked serialization dll, thus avoiding any reflection / analysis / compilation at runtime. – Marc Gravell Apr 30 '13 at 18:01
  • Thanks. It is indeed a false positive and did not re-occur. (It is a throw-away program so I'll do nothing. Just wanted to let you know that there is no bug/actual deadlock). – usr Apr 30 '13 at 18:09
  • We see this come up very infrequently (three times over the last two years). When it does, it cripples our website. Any updates or recommendations @MarcGravell ? – AgileJon Aug 11 '14 at 20:34
  • 1
    @AgileJon if you want to eliminate it completely, you could use the separate precompiler exe to generate a serialization dll that then never needs to do any metadata inspection – Marc Gravell Aug 11 '14 at 21:07
  • @MarcGravell any updates on this? I am getting the exception in a heavily parallelised serialisation routine (16 cores doing huge amounts of serialisation) at random times, there is no deadlock however, just the exception. "you could use the separate precompiler exe to generate a serialization dll" Is this still a valid comment? I have just adjusted my timeout to 30,000ms to see if that helps. – rollsch Aug 10 '21 at 06:49
1

I'm getting this same exact error on my servers. Though I'm not sure what is causing the error. A few days ago it happened twice a few hours apart while our servers were under the highest load we've experienced. Running 8 servers the CPU across all of them went from 70% to 100% in seconds but a slightly different times. For example each of the servers might have started this spike 1-5 minutes after the first.

Never seen this happen before and i've had this code in production for a few months. Still haven't been able to reproduce it and I can't tell if the error is thrown because the server was at 100% CPU or if this is what caused the server to spike. Stopping all the connections to the servers and letting the cpu drop back to 0 fixed the issue. No iis restart was required.

When IIS starts i run the following code once for each type.

var type = this.GetType();
RuntimeTypeModel.Default.Add(type, true);
Int32 i = 1;
foreach(PropertyInfo info in type.GetProperties())
{
    if(info.CanWrite)
    {
        RuntimeTypeModel.Default[type].AddField(i++, info.Name);
    }
}
bwight
  • 3,300
  • 17
  • 21