How to writte protobuf serialized content directly to SharpZipLib stream

Question

Is there possible to writte protobuf serialized content directly to SharpZipLib stream? When I try to do this, looks like the provided stream is not filled with the data from protobuf. Later I would need to get back the deserialized entity from provided Zip stream. My code looks like this:

private byte[] ZipContent(T2 content)
    {
        const short COMPRESSION_LEVEL = 4; // 0-9
        const string ENTRY_NAME = "DefaultEntryName";


        byte[] result = null;

        if (content == null)
            return result;

        IStreamSerializerProto<T2> serializer = this.GetSerializer(content.GetType());
        using (MemoryStream outputStream = new MemoryStream())
        {
            using (ZipOutputStream zipOutputStream = new ZipOutputStream(outputStream))
            {
                zipOutputStream.SetLevel(COMPRESSION_LEVEL);
                ZipEntry entry = new ZipEntry(ENTRY_NAME);
                entry.DateTime = DateTime.Now;
                zipOutputStream.PutNextEntry(entry);

                serializer.Serialize(zipOutputStream, content);
            }

            result = outputStream.ToArray();
        }

        return result;
    }

private class ProtobufStreamSerializer<T3> : IStreamSerializerProto<T3>
        {
            public ProtobufStreamSerializer()
            {
                ProtoBuf.Serializer.PrepareSerializer<T3>();
            }

    public void Serialize(Stream outputStream, T3 content)
    {
        Serializer.Serialize(outputStream, content);
    }

            public T3 Deserialize(Stream inputStream)
            {
                T3 deserializedObj;
                using (inputStream)
                {
                    deserializedObj = ProtoBuf.Serializer.Deserialize<T3>(inputStream);
                }
                return deserializedObj;
            }
        }

Sample of a class which I'm trying to serialize:

[Serializable]
    [ProtoContract]
    public class Model
    {
        [XmlElement("ModelCode")]
        [ProtoMember(1)]
        public int ModelCode { get; set; }

                ...

    }

what is the data you are serializing? i.e. what is `content`? What is the type's definition (ideally, so I can see the attributes), and what are the values associated with the instance? — Marc Gravell, Oct 25 '13 at 13:32
Your edit still isn't enough to make me think that 0 bytes is unexpected. If the `Model` is all default values, there is nothing interesting to write, so it is entirely possible to write it in 0 bytes. Do you have a complete example that shows **actual data** that we might reasonably expect to take space? — Marc Gravell, Oct 25 '13 at 13:47
Note that compression of binary data is not a "no-brainer"; it can often make the output **larger**. Caution is advised. — Marc Gravell, Oct 25 '13 at 13:59

Jon Skeet · Accepted Answer · 2013-10-25T13:44:02.477

4

This is the problem, I believe (in the original code in the question):

public void Serialize(Stream outputStream, T3 content)
{
    using (var stream = new MemoryStream())
    {
        Serializer.Serialize(stream, content);
    }
}

You're completely ignoring outputStream, instead writing the data just to a new MemoryStream which is then ignored.

I suspect you just want:

public void Serialize(Stream outputStream, T3 content)
{
    Serializer.Serialize(outputStream, content);
}

I'd also suggest removing the using statement from your Deserialize method: I'd expect the caller to be responsible for disposing of the input stream when they're finished with it. Your method can be simplified to:

public T3 Deserialize(Stream inputStream)
{
    return ProtoBuf.Serializer.Deserialize<T3>(inputStream);
}

edited Oct 25 '13 at 13:44

answered Oct 25 '13 at 13:21

Jon Skeet

1,421,763
867
9,128
9,194

Yes, You're right, but still before executing that line and atfer, the stream has the same length, so looks like nothing happened. – vts123 Oct 25 '13 at 13:25
1

@Vytas999 well, what is in `content`? 0 bytes is a perfectly valid length for a protobuf message... **if** there is nothing interesting to write – Marc Gravell Oct 25 '13 at 13:30
There is a Class containing a lot of properties with values. I edit the question with details about how it looks. – vts123 Oct 25 '13 at 13:37
1

@Vytas999: Having changed the question, my answer now looks non-sensical... it would be nice if you'd at least acknowledged that change in the question. It looks like it should be okay though - if you serialize to a `MemoryStream` instead of a `ZipOutputStream`, what happens then? Do you perhaps need to explicitly close each entry with `ZipOutputStream`? – Jon Skeet Oct 25 '13 at 13:43
Thanks Jon, Your suggestion was right, I changed the method signature to public void Serialize(ZipOutputStream outputStream, T3 content) and public T3 Deserialize(ZipInputStream inputStream) and it worked now when serializing/deserializing to/from Zip stream. – vts123 Oct 25 '13 at 13:53
1

@Vytas999: You really shouldn't have needed to change the parameter type at all. I'd be very surprised if that actually made a difference. Perhaps you hadn't got a properly clean build after the first change? – Jon Skeet Oct 25 '13 at 20:19
@Jon Skeet: Again, You're right. After clean rebuild it works wit the more general "Stream" too. Thanks. – vts123 Oct 28 '13 at 12:00

score 2 · Answer 2 · answered Oct 25 '13 at 13:46

The code (with the edit pointed out by Jon) looks fine. Here it is working:

static void Main()
{
    var obj = new Bar{ X = 123, Y = "abc" };
    var wrapper = new Foo<Bar>();
    var blob = wrapper.ZipContent(obj);
    var clone = wrapper.UnzipContent(blob);
}

where Bar is:

[ProtoContract]
class Bar
{
    [ProtoMember(1)]
    public int X { get; set; }
    [ProtoMember(2)]
    public string Y { get; set; }
}

and Foo<T> is your class (I didn't know the name), where I have added:

public T2 UnzipContent(byte[] data)
{
    using(var ms = new MemoryStream(data))
    using(var zip = new ZipInputStream(ms))
    {
        var entry = zip.GetNextEntry();
        var serializer = this.GetSerializer(typeof(T2));
        return serializer.Deserialize(zip);
    }
}

Also, note that compression is double-edged. In the example I give above, the underlying size (i.e. if we just write to a MemoryStream) is 7 bytes. ZipInputStream "compresses" this 7 bytes down to 179 bytes. Compression works best on larger objects, usually when there is lots of text content.

How to writte protobuf serialized content directly to SharpZipLib stream

2 Answers2