GZipStream works when writing to FileStream, but not MemoryStream

Question

If compress some json text, and write that it to a file using a FileStream I get the expected results. However, I do not want to write to disk. I simply want to memorystream of the compressed data.

Compression to FileStream:

string json = Resource1.json;

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (FileStream output = File.Create(@"C:\Users\roarker\Desktop\output.json.gz"))
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);
    }
}

Above works. Below, the output memory stream is length 10 and results in an empty .gz file.

string json = Resource1.json;

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (MemoryStream output = new MemoryStream())
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);

        byte[] bytes = output.ToArray();
    }
}

EDIT: Moving output.ToArray() outside the inner using clause seems to work. However, this closes the output stream for most usage. IE:

        using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
        using (MemoryStream output = new MemoryStream())
        {
            using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
            {
                input.CopyTo(compression);
            }
            WriteToFile(output);
        }

where :

    public static void WriteToFile(Stream stream)
    {
        using (FileStream output = File.Create(@"C:\Users\roarker\Desktop\output.json.gz"))
        {
            stream.CopyTo(output);
        }
    }

This will fail on stream.CopyTo because the stream has been closed. I know I could make a new Stream from bytes of output.ToArray(), but why is this necessary? why does ToArray() work when the stream is closed?

Final Edit:

Just needed to use the contructor of the GZipStream with the leaveOpen parameter.

Do you get the same behavior if you move `byte[] bytes = output.ToArray();` outside the inner `using` block? — GalacticCowboy, Jan 13 '16 at 19:47
Streams are like toilets. You're not done with them until they're flushed. Unless you're some kind of disgusting animal. I mean, really. — , Jan 13 '16 at 19:55
Related: https://stackoverflow.com/questions/34566559/is-it-possible-to-get-length-of-closed-memory-stream/34566825#34566825 — H H, Jan 13 '16 at 21:15

score 17 · Accepted Answer · answered Jan 13 '16 at 19:50

17

You're calling ToArray() before you've closed the GZipStream... that means it hasn't had a chance to flush the final bits of its buffer. This is a common issue for compression an encryption streams, where closing the stream needs to write some final pieces of data. (Even calling Flush() explicitly won't help, for example.)

Just move the ToArray call:

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (MemoryStream output = new MemoryStream())
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);
    }
    byte[] bytes = output.ToArray();
    // Use bytes
}

(Note that the stream will be disposed when you call ToArray, but that's okay.)

answered Jan 13 '16 at 19:50

Jon Skeet

1,421,763
867
9,128
9,194

I'm confused because the output stream is considered closed after the inner using. I can use ToArray() and get all the bytes, but I cannot pass the `output` to other function to be treated like a memorystream. Does this make sense? – Dave Jan 13 '16 at 20:33
@Dave: Yes - basically you're using the extra functionality of `MemoryStream` (that it has the data in memory) but you can't use it as a regular stream. It's really, really handy to be able to do this - precisely in this sort of situation. – Jon Skeet Jan 13 '16 at 20:33
I edit'd the question, but I guess what you're saying is that if I want to access that data in memory, I need to copy those bytes into a new stream? – Dave Jan 13 '16 at 20:44
@Dave: No, I'm not saying that at all. You call `ToArray()`, and you get a byte array with the contents. Access that however you want to... there's no need to create a new stream with that data unless you want to. If you *need* it in the form of a stream, then yes, you'd create a new `MemoryStream` to wrap it. In some cases there are ways to make "wrapping" streams not close the underlying ones, but it's not universal. – Jon Skeet Jan 13 '16 at 20:45
2

Ahh, I can probably use the leaveOpen constructor arguement to do what I want. – Dave Jan 13 '16 at 20:49
I thought the whole point of working with Streams is so you don't have to create arrays (which chew up memory). Surely it's possible to pass in a stream and compress the contents to another stream. In my case I need a stream to pass to another method which "streams" the data to azure storage - ideally there would be no arrays in that process. – tigerprawn May 25 '21 at 14:46
@tigerprawn: We can't really tell much about your context here. This answer is about arrays because the *question* is about arrays. Presumably in your context, you *don't* have arrays. If you're having difficulty with what you're trying to do, I suggest you ask a new question with a lot more detail. – Jon Skeet May 25 '21 at 14:51

GZipStream works when writing to FileStream, but not MemoryStream

1 Answers1