After deploying a new version of one of our applications we started seeing multiple ArgumentOutOfRangeExceptions from WebApiContrib/protobuf-net. It had happened once before, but occured more frequently (5-10 times or so) after the deploy. It only happened on a few of the total requests and I have not been able to reproduce this locally or in other environments. The errors occured on multiple machines.
The calling applications install a Nuget class library with the protobuf contract, and this contract has been updated in the called application in this release. Some properties got removed (not changing ProtoMember.Tag) and some properties got marked as IsRequired=false instead of true.
Does anyone have any insights to what may be failing or how we can diagnose it further?
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: offset
at System.Web.HttpInputStream.Seek(Int64 offset, SeekOrigin origin)
at System.Web.Http.WebHost.SeekableBufferedRequestStream.Seek(Int64 offset, SeekOrigin origin)
at ProtoBuf.ProtoReader.Seek(Stream source, Int32 count, Byte[] buffer) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 1143
at proto_14(Object , ProtoReader )
at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 579
at proto_2(Object , ProtoReader )
at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 704
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 588
at WebApiContrib.Formatting.ProtoBufFormatter.ReadFromStreamAsync(Type type, Stream stream, HttpContent content, IFormatterLogger formatterLogger)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext()
Versions:
<package id="Microsoft.AspNet.WebApi" version="5.2.3" targetFramework="net452" />
<package id="protobuf-net" version="2.0.0.668" targetFramework="net452" />
<package id="WebApiContrib.Formatting.ProtoBuf" version="0.9.5.0" targetFramework="net452" />
Update:
- We tried removing the ProtobufFormatter from WebApiContrib.Formatting and instead doing the deserialization directly in code with Serializer.Deserialize. That had no effect.
- Then we added retries. Resetting stream.Position to 0 and deserializing again. That works.
- We are now logging the Request.Content as a base64 string. That content deserializes just fine and when trying to reproduce it locally, nothing fails, even with thousands of requests.
