Send/Receive GZip compressed MSMQ messages in C#

Question

I am trying to send large objects (>30MB) to a MSMQ queue. Due to the large amount of data we are are tring to send the idea was to GZip the objects prior to sending them, then unzipping them on the receiving end.

However, writing the compressed stream to the message.BodyStream property seems to work, but not reading it out from there. I don't know what's wrong.

Message l_QueueMessage = new Message();
l_QueueMessage.Priority = priority;

using (MessageQueue l_Queue = CreateQueue())
{
    GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress);

    Formatter.Serialize(stream, message);

    l_Queue.Send(l_QueueMessage);
}

The Formatter is a global property of type BinaryFormatter. This is used to serialize/deserialize to the type of object we want to send/receive, e.g. "ProductItem".

The receving end looks like this:

GZipStream stream = new GZipStream(l_Message.BodyStream, CompressionMode.Decompress);

object decompressedObject = Formatter.Deserialize(stream);

ProductItem l_Item = decompressedObject as ProductItem;

m_ProductReceived(sender, new MessageReceivedEventArgs<ProductItem>(l_Item));

l_ProductQueue.BeginReceive();

I get an EndOfStreamException "{"Unable to read beyond the end of the stream."} trying to deserialize at System.IO.BinaryReader.ReadByte()

Using the messageBodyStream property I actually circumvent the message.Formatter, which I don't initialize to anything, becasue I'm using my own ser/deser mechanism with the GZipStream. However, I am not sure if that's the correct way of doing this.

What am I missing? Thanks!

score 3 · Answer 1 · answered Aug 29 '11 at 10:28

3

In your original code, the problem is that you need to close the GZipStream in order for a GZip footer to be written correctly, and only then you can send it. If you dont, you end up sending bytes that can not be deserialized. That's also why your new code where sending is done later works.

answered Aug 29 '11 at 10:28

Tsabo

834
4
9

Thanks a lot for that observation. I was wondering if the gzip stream would need to be opened/closed and whether that affects the decompressing. In either case, I wonder why - using my old receiving code - would not work even with the new sender code I posted above. – John Aug 29 '11 at 10:49
I can confirm that even though I now close the gzip stream correctly before sending, I still can't deserialize. – John Aug 29 '11 at 11:18
Still the main part of solution was in the receiver's code. GZipStream was closed before sending in my example as well and as stated by John it didn't really work. – rudolf_franek Aug 29 '11 at 11:41

John · Accepted Answer · 2011-08-31T12:22:42.877

OK, I made this work. The key was to convert the decompressed stream on the receiver to a byte[] array. Then the deserialization started working.

The sender code (notice the stream is closed before sending the message):

using (MessageQueue l_Queue = CreateQueue())
{
    using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Compress, true))
    {
        Formatter.Serialize(stream, message);
    }

    l_Queue.Send(l_QueueMessage);
}

The receiving end (notice how I convert the stream to a byte[] then deserialize):

using (GZipStream stream = new GZipStream(l_QueueMessage.BodyStream, CompressionMode.Decompress))
{
    byte[] bytes = ReadFully(stream);

    using (MemoryStream ms = new MemoryStream(bytes))
    {
        decompressedObject = Formatter.Deserialize(ms);
    }
}

Still, don't know why this works using the ReadFully() function and not the Stream.CopyTo(). Does anyone?

Btw, ReadFully() is a function that creates a byte[] out of a Stream. I have to credit Jon Skeet for this at http://www.yoda.arachsys.com/csharp/readbinary.html. Thanks!

rudolf_franek · Answer 3 · 2011-08-29T11:34:27.587

1

Try to separate compressing and sending:

byte[] binaryBuffer = null;
using (MemoryStream compressedBody = new MemoryStream()) 
{
    using(GZipStream stream = new GZipStream(compressedBody, CompressionMode.Compress))
    {
        Formatter.Serialize(compressedBody, message); 
        binaryBuffer = compressedBody.GetBuffer();
    }
}

using (MessageQueue l_Queue = CreateQueue())        
{            
    l_QueueMessage.BodyStream.Write(binaryBuffer, 0, binaryBuffer.Length);
    l_QueueMessage.BodyStream.Seek(0, SeekOrigin.Begin);
    l_Queue.Send(l_QueueMessage);
}

edited Aug 29 '11 at 11:34

answered Aug 29 '11 at 08:15

rudolf_franek

1,795
3
28
41

Thanks for your time. I tried the code you suggested but no luck. Sending is fine, apparently, but the receiving end is not working. Using my original code I get a "magic packet" exception, which means the message body stream is not a GZip stream. "The magic number in GZip header is not correct. Make sure you are passing in a GZip stream." – John Aug 29 '11 at 09:32
You can only write to buffer after GZipStrem is closed. – Tsabo Aug 29 '11 at 10:32

Send/Receive GZip compressed MSMQ messages in C#

3 Answers3