3

I have a transport application which is used in asa a pub-/sub server to relay data between clients. The pub-/sub server need only know a little about each piece of data, for example it needs the topicname to be able to relay a published topic to the correct subscribers.

To achieve this I have thought out a scheme in which a class that is decorated as a ProtoContract includes a byte[] which in turn contains protobuf-net serialized data. This way the server does only have to deserialize a very small part of the data it relays (and does not need to know types). My type looks like this;

[ProtoContract]
public class DataFrame
{
    /// <summary>
    /// Time the data was issued or sampled. In UTC time.
    /// </summary>
    [ProtoMember(1)]
    public DateTime TimeStamp = DateTime.UtcNow;

    /// <summary>
    /// The topic associated with the data
    /// </summary>
    [ProtoMember(2)]
    public string TopicName = string.Empty;

    /// <summary>
    /// Command, can be either Discover, Subscribe, Unsubscribe or Publish
    /// </summary>
    [ProtoMember(3)]
    public string Command = string.Empty;

    /// <summary>
    /// The fully qualified type name of the content
    /// </summary>
    [ProtoMember(4)]
    private string _typeName = string.Empty;

    /// <summary>
    /// Serialized content data (if any)
    /// </summary>
    [ProtoMember(5)]
    private byte[] _content;

    /// <summary>
    /// The fully qualified type name of the content
    /// </summary>
    public string TypeName
    {
        get
        {
            return _typeName;
        }
    }

    /// <summary>
    /// Get the content of this DataFrame
    /// </summary>
    /// <typeparam name="T">Type of the content</typeparam>
    /// <returns>The content</returns>
    public T GetContent<T>()
    {
        MemoryStream ms = new MemoryStream(_content);
        return Serializer.DeserializeWithLengthPrefix<T>(ms, PrefixStyle.Base128);
    }

    /// <summary>
    /// Set the content for this DataFrame
    /// </summary>
    /// <param name="value">The content to set, must be serializable and decorated as a protobuf contract type</param>
    public void SetContent<T>(T value)
    {
        MemoryStream ms = new MemoryStream();
        Serializer.SerializeWithLengthPrefix(ms, value, PrefixStyle.Base128);
        _content = ms.GetBuffer();
        _typeName = value.GetType().AssemblyQualifiedName;
    }

    /// <summary>
    /// Encode the frame to a serialized byte array suitable for tranmission over a network
    /// </summary>
    /// <returns>The encoded byte[]</returns>
    public byte[] Encode()
    {
        DataFrame frame = (DataFrame)this;
        MemoryStream ms = new MemoryStream();
        Serializer.SerializeWithLengthPrefix(ms, frame, PrefixStyle.Base128);
        return ms.GetBuffer();
    }

    /// <summary>
    /// Factory function to create a frame from a byte array that has been received
    /// </summary>
    /// <param name="buffer">The serialized data to decode</param>
    /// <returns>A new dataframe decoded from the byte[]</returns>
    public static DataFrame Decode(byte[] buffer)
    {
        MemoryStream ms = new MemoryStream(buffer);
        DataFrame frame = Serializer.DeserializeWithLengthPrefix<DataFrame>(ms, PrefixStyle.Base128);
        frame._timeStamp = DateTime.SpecifyKind(frame._timeStamp, DateTimeKind.Utc);
        return frame;
    }
}

Problem is, that I am able to deserialized a DataFrame, but when deserializing the payload byte[] I get protobuf exceptions. That is, this works (server is a UdpClient);

data = server.Receive(ref remoteEP);
DataFrame frame = DataFrame.Decode(data);

But this will give me a protobuf exception, even if the content is a string;

string content = frame.GetContent<string>();

Does anyone have any pointers on what I am doing wrong?

par.degerman
  • 41
  • 1
  • 4
  • Note: that usage of `GetBuffer()` is incorrect - you will be transferring too much data (up to twice as much, worst-case - for both the inner and outer array separately - so more than 2x is theoretically possible); you should really use `ToArray()` instead in this case; since you are using `*WithLengthPrefix`, this is unlikely to cause it to fail, though (just to be inefficient). What is the actual "protobuf exception" that it raises? – Marc Gravell Aug 15 '13 at 11:46
  • This works fine: http://pastie.org/8238757 (data was 370 bytes using `GetBuffer()`, and 118 bytes using `ToArray()`, but both worked) - can you perhaps give me more context in terms of something I can do to reproduce the error you are seeing? Have you checked that the data you received is 100% identical to the data you thought you sent? – Marc Gravell Aug 15 '13 at 11:51
  • Oh, and btw: 90 bytes of that 118 was the `AssemblyQualifiedName` of `string`... that can be a pretty expensive way of specifing the type – Marc Gravell Aug 15 '13 at 11:53
  • After checking and rechecking I saw that the error actually was due to a bug in how I reassembled UDP packets. It only showed its ugly head on larger packets, so my inefficient code triggered the bug. I changed the code according to your suggestions of using ToArray() instead and it really saves me a lot of wire space. – par.degerman Aug 16 '13 at 08:01
  • btw. one of my content types contained a byte[]. This seems to grow in size after a serializing and de-serializing sequence. I.e. if I have a byte[8] on one side I get a byte[16] on the other. The original bytes seem to reside in the last half of the byte[16]. I saw on another post ([link](http://stackoverflow.com/questions/9142819/protobuf-net-overwritelist-on-byte-array)) that this was an old bug that should have been fixed now. I'm using version 2.0.0.640 from NUget. – par.degerman Aug 16 '13 at 08:07
  • 1
    I'm going to guess that you initialize the field to `new byte[8]` in the constructor / field-initialize, right? If so, either suppress the constructor (`[ProtoContract(SkipConstructor=true)]`) or suppress append (`[ProtoMember(n, OverWriteList=true)]`). protobuf is designed (by google) as concat===append. – Marc Gravell Aug 16 '13 at 08:43

0 Answers0