31

Our system, when serializing one message using protobuf-net, sometimes, but not every time, raises the error exposed below. What are the reasons for the error and how can I mitigate it?

Please notice that we are using DeserializeWithLengthPrefix already.

UPDATE: the relevant code is here

private const PrefixStyle PrefixStyleInPlace = PrefixStyle.Fixed32;
public static byte[] SerializeObjectToByteArray<TSerializable>(TSerializable source) where TSerializable : class
    {
        byte[] result;
        using (var memoryStream = SerializeObjectToStream(source))
        {
            result = memoryStream.ToArray();
        }
        return result;
    }


    public static TResult DeserializeObject<TResult>(byte[] source)
    {
        TResult result;
        using (var memoryStream = new MemoryStream(source))
        {
            memoryStream.Position = 0;
            result = Serializer.DeserializeWithLengthPrefix<TResult>(memoryStream,PrefixStyleInPlace);
        }

        return result;
    }

    public static MemoryStream SerializeObjectToStream<TSerializable>(TSerializable source) where TSerializable : class
    {
        var memoryStream = new MemoryStream();

        Serializer.SerializeWithLengthPrefix(memoryStream, source,PrefixStyleInPlace);
        memoryStream.Position = 0;
        return memoryStream;
    }


    public static TResult DeserializeObject<TResult>(MemoryStream sourceStream)
    {
        TResult result;
        result = DeserializeObject<TResult>(sourceStream.ToArray());
        return result;
    }

MESSAGE:

System.IO.EndOfStreamException : Attempted to read past the end of the
 stream.
 +++++++++++++++++++ STACK TRACE: at ProtoBuf.ProtoReader.Ensure(Int32 count, Boolean trict) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 234 at
 ProtoBuf.ProtoReader.ReadString() in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 471 at
 proto_15(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_16(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_11(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_16(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_13(Object , ProtoReader ) at
ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_16(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key,
 ProtoReader reader, Type type) in
 c:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 556 at
 proto_2(Object , ProtoReader ) at
 ProtoBuf.Serializers.CompiledSerializer.ProtoBuf.Serializers.IProtoSerializer.Read(Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Serializers\CompiledSerializer.cs:line
 49 at ProtoBuf.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object
 value, ProtoReader source) in
 c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 721 at
 ProtoBuf.Meta.TypeModel.DeserializeWithLengthPrefix(Stream source,
 Object value, Type type, PrefixStyle style, Int32 expectedField,
 TypeResolver resolver, Int32& bytesRead, Boolean& haveObject,
 SerializationContext context) in
 c:\Dev\protobuf-net\protobuf-net\Meta\TypeModel.cs:line 351 at
 ProtoBuf.Serializer.DeserializeWithLengthPrefix[T](Stream source,
 PrefixStyle style, Int32 fieldNumber) in
 c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 303 at
 ProtoBuf.Serializer.DeserializeWithLengthPrefix[T](Stream source,
 PrefixStyle style) in
 c:\Dev\protobuf-net\protobuf-net\Serializer.cs:line 288 at
 ermeX.Common.ObjectSerializer.DeserializeObject[TResult](Byte[]
 source) in
    [ProtoContract(SkipConstructor = true)]
    [ProtoInclude(100, typeof(BusMessage))]
    [ProtoInclude(200, typeof(TransportMessage))]
    [ProtoInclude(300, typeof(BizMessage))]

Code here

 internal abstract class SystemMessage : ISystemMessage, IEquatable<SystemMessage>
    {
        protected SystemMessage():this(Guid.NewGuid(),DateTime.UtcNow)
        {
        }
        protected SystemMessage(Guid messageId,DateTime createdTimeUtc)
        {
            MessageId = messageId;
            CreatedTimeUtc = new DateTime(createdTimeUtc.Ticks);//TODO: UNTIL PROTOBUF-NET FIXES ISSUE 335 
        }

        [ProtoMember(1)]
        public Guid MessageId{get;private set;}

        [ProtoMember(2)]
        public DateTime CreatedTimeUtc { get; private set; }

        ...
    }


    [ProtoContract(SkipConstructor = true)]
    internal sealed class TransportMessage : SystemMessage, ISystemMessage<BusMessage>
    {
        //just for the serializer, remove in the future
        private TransportMessage()
        {
        }

        public TransportMessage(Guid recipient, BusMessage data)
            : this(data.MessageId, data.CreatedTimeUtc, recipient, data)
        {
        }

        public TransportMessage(Guid messageId, DateTime createdTimeUtc, Guid recipient, BusMessage data)
            : base(messageId, createdTimeUtc)
        {
            if (data == null) throw new ArgumentNullException("data");
            if (recipient.IsEmpty()) throw new ArgumentException("recipient cannot be an empty value");
            Recipient = recipient;
            Data = data;
        }

        [ProtoMember(1)]
        public Guid Recipient { get; private set; }

        [ProtoMember(2)]
        public BusMessage Data { get; private set; }
    }
    [ProtoContract(SkipConstructor = true)]
    internal sealed class BusMessage: SystemMessage, ISystemMessage<BizMessage>,IEquatable<BusMessage>
    {
        private BusMessage()
        {

        }
        public BusMessage(Guid publisher,BizMessage data)
            : this(data.MessageId,data.CreatedTimeUtc,publisher, data)
        {


        }

        public BusMessage(Guid messageId, DateTime createdTimeUtc, Guid publisher, BizMessage data) : base(messageId,createdTimeUtc)
        {
            if (data == null) throw new ArgumentNullException("data");
            Publisher = publisher;
            Data = data;
        }
        [ProtoMember(1)]
        public Guid Publisher { get; protected set; }

        [ProtoMember(2)]
        public BizMessage Data { get; protected set; }
    }

    [ProtoContract(SkipConstructor = true)]
    internal sealed class BizMessage : SystemMessage, IEquatable<BizMessage>
    {
        private string _jsonMessage;
        private object _data = null;

        public BizMessage(object data) : base()
        {
            if (data == null) throw new ArgumentNullException("data");
            _data = data;
        }

        private BizMessage(){}

        public static BizMessage FromJson(string jsonData)
        {
            if(string.IsNullOrEmpty(jsonData)) 
                throw new ArgumentException();
            return new BizMessage(){JsonMessage = jsonData};
        }

        [ProtoMember(75)]
        internal string JsonMessage
        {
            get
            {
                if(string.IsNullOrEmpty(_jsonMessage))
                {
                    if (_data == null)
                        throw new ApplicationException(
                            "One of both, _data or the serialized json message must have a value");
                    _jsonMessage = JsonSerializer.SerializeObjectToJson(_data);
                }
                return _jsonMessage;
            }
            private set { _jsonMessage = value; }
        }

        public Type MessageType
        {
            get
            {
                UpdateData();
                if (_data == null)
                    return typeof(void);
                return _data.GetType();
            }
        }

        public object RawData
        {
            get
            {
                UpdateData();
                return _data;
            }
        }

        private void UpdateData()
        {
            if (_data == null)
            {
                if (string.IsNullOrEmpty(_jsonMessage))
                    throw new ApplicationException(
                        "One of both, _data or the serialized json message must have a value");
                _data = JsonSerializer.DeserializeObjectFromJson<object>(_jsonMessage);
            }
        }
    }
Sitansu
  • 3,225
  • 8
  • 34
  • 61
Mimere
  • 763
  • 2
  • 10
  • 15
  • Any chance I can see the code where you serialize / deserialize? – Marc Gravell Jan 14 '13 at 18:32
  • interesting. I'll have to take a look tomorrow. End of a long day. – Marc Gravell Jan 15 '13 at 20:00
  • I've had a look with some (admittedly simple) test models, and I simple can't get it to mis-behave; it just works. Is there anything I can look at that might help me reproduce the issue? – Marc Gravell Jan 16 '13 at 10:13
  • I have updated the ticket with the definitions im serializing – Mimere Jan 16 '13 at 11:22
  • ah, right - will look; small request: can you add `@Marc` when you reply, so that the system tells me ;p (since you're the post owner, I don't need to add `@Miguel` - it assumes you're interested) – Marc Gravell Jan 16 '13 at 13:34
  • 4
    I've tested with [this test](http://pastie.org/5696364), and it runs successfully to completion (10M iterations): I genuinely want to help, but I'm pretty limited since I can't reproduce the error. Does that test work successfully for you? – Marc Gravell Jan 16 '13 at 13:48
  • @MarcGravell Yes Marc, the test works as well as my current tests. But as said, this error is raised sometimes. I am also serializing/deserializing from/to a network stream. The issue might be there. – Mimere Jan 16 '13 at 14:22
  • well, in theory there's no difference... However: is this sending multiple separate messages after each other on the same stream? If so, I can try tweaking the test to be comparable to that... – Marc Gravell Jan 16 '13 at 20:24
  • @MarcGravell, This happens in one test where the object contained by the BizMessage contains a byte[] property could have 10MB. Any tip? – Mimere Jan 17 '13 at 20:54
  • well, that *should* work fine, but indeed that is an interesting case for me to try to look at. I have struggled to repro so far. – Marc Gravell Jan 17 '13 at 21:02
  • 1
    Which version of proto-buf are you using? – Swen Kooij Aug 14 '13 at 15:41
  • 3
    I wonder if you looked to check if there might be a concurrency issue, somewhere in you app. In most cases when you get intermittent issues with streams is due to the fact that they are being concurrently accessed and are not thread safe – Dmitry Dec 05 '13 at 03:05

2 Answers2

5

I've seen this error when the "type" that I specify to deserialize to is not the same as the originating type that was serialized. I suppose this could also happen if your object has changed and you're attempting to deserialize an older version of your object (i.e. old byte array) that is incompatible with the new version.

Brian Booth
  • 727
  • 1
  • 6
  • 10
0

When I used socket transmission, this error occurred. I found that it was caused by incomplete data.
The server sends the packet length of 1024 bytes to the client, but the client only receives 800 bytes. There is an error in parsing.

jasie
  • 2,192
  • 10
  • 39
  • 54
yang
  • 31
  • 1
  • 3
  • Please elaborate on 'imcomplete data'. Your answer is very scarce. – jasie Sep 22 '20 at 08:01
  • The server sends the packet length of 1024 bytes to the client, but the client only receives 800 bytes. There is an error in parsing – yang Sep 26 '20 at 06:11