XmlSerialize directly to GZipStream throws magic number exception on Decompression

Question

I am trying to Serialize an object to XML however my object is a generic list containing many records and causes the serializer to consume lots of memory. So I tried to serialize directly to a GZipStream with the following code:

Dim formatter As XmlSerializer = XmlSerializerFactory.GetSerializerForType(_type)
Using _ms As New MemoryStream()
    Using gzStream As New GZipStream(_ms, CompressionMode.Compress, True)

        _ms.Position = 0
        formatter.Serialize(gzStream, obj)
        _ms.Position = 0
        gzStream.Flush()
        gzStream.Close()
    End Using

    _ms.Position = 0
    Dim decompressData() As Byte
    Using gzStream As New GZipStream(_ms, CompressionMode.Decompress)
        ReDim decompressData(9000 - 1) 'this number doesn't matter, the data in my test sample is small
        Dim Len As Integer = gzStream.Read(decompressData, 0, decompressData.Length)
    End Using
End Using

However I run into an InvalidDataException The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. when trying to read the data into the decompressData array.

When I Serialize to a separate memory stream first and then compress that stream such as:

Dim formatter As XmlSerializer = XmlSerializerFactory.GetSerializerForType(_type)
Using _ms As New MemoryStream()
    Dim uc_fileBytes() As Byte
    Dim uc_len As Integer
    Using _ms101 As New MemoryStream()
        formatter.Serialize(_ms101, obj)
        uc_fileBytes = _ms101.GetBuffer()
        uc_len = _ms101.Length
    End Using
    Using gzStream As New GZipStream(_ms, CompressionMode.Compress, True)
        _ms.Position = 0
        gzStream.Write(uc_fileBytes, 0, uc_len)
        gzStream.Flush()
        gzStream.Close()
    End Using

    Dim decompressData() As Byte
    Using gzStream As New GZipStream(_ms, CompressionMode.Decompress)
        ReDim decompressData(9000 - 1)
        Dim Len As Integer = gzStream.Read(decompressData, 0, decompressData.Length)
    End Using
End Using

It works fine without error. But why does it fail when I serialize directly to the GZipStream?

Answer 1

The cause of the problem is because the GZipStream behaves differently (obviously) to the MemoryStream when writing to it. It doesn't handle paged writes very well.