2

Can I get a GZipStream for a file on disk without writing the entire compressed content to temporary storage? I'm currently using a temporary file on disk in order to avoid possible memory exhaustion using MemoryStream on very large files (this is working fine). 

    public void UploadFile(string filename)
    {
        using (var temporaryFileStream = File.Open("tempfile.tmp", FileMode.CreateNew, FileAccess.ReadWrite))
        {   
            using (var fileStream = File.OpenRead(filename))
            using (var compressedStream = new GZipStream(temporaryFileStream, CompressionMode.Compress, true))
            {
                fileStream.CopyTo(compressedStream);
            }

            temporaryFileStream.Position = 0;

            Uploader.Upload(temporaryFileStream);
        }
    }

What I'd like to do is eliminate the temporary storage by creating GZipStream, and have it read from the original file only as the Uploader class requests bytes from it. Is such a thing possible? How might such an implementation be structured?

Note that Upload is a static method with signature static void Upload(Stream stream).

Edit: The full code is here if it's useful. I hope I've included all the relevant context in my sample above however.

Boinst
  • 3,365
  • 2
  • 38
  • 60
  • 1
    How do you upload? Do you use a *NetworkStream*? Or do you use a *HttpWebClient*? Or something else? –  Jun 17 '14 at 00:17
  • @elgonzo I use AWS SDK TransferUtilityUploadRequest. The full code is [here](http://pastebin.com/Z7disr1F) if it's useful although I expect that it is not. I do not have control over the code that consumes the `Stream`, I just have to provide it with a `Stream` object. – Boinst Jun 17 '14 at 00:23
  • You would need to replace temporaryFileStream in the GZipStream constructor with an stream to upload to. – Mike Hixson Jun 17 '14 at 00:24
  • @elgonzo that doesn't quite work because the `GZipStream` doesn't contain the bytes until you write them in. I want to start reading from the compressed stream before I finish writing to it, and I don't want to hold the entire contents of the compressed stream either on disk or in memory while I do it. – Boinst Jun 17 '14 at 00:30
  • @Boinst, sorry, i was wrong. Didn't notice that GZipStream doesn't work as filter stream for reading an uncompressed file and spitting out compressed data... Doh... –  Jun 17 '14 at 00:40

1 Answers1

5

Yes, this is possible, but not easily with any of the standard .NET stream classes. When I needed to do something like this, I created a new type of stream.

It's basically a circular buffer that allows one producer (writer) and one consumer (reader). It's pretty easy to use. Let me whip up an example. In the meantime, you can adapt the example in the article.

Later: Here's an example that should come close to what you're asking for.

using (var pcStream = new ProducerConsumerStream(BufferSize))
{
    // start upload in a thread
    var uploadThread = new Thread(UploadThreadProc(pcStream));
    uploadThread.Start();

    // Open the input file and attach the gzip stream to the pcStream
    using (var inputFile = File.OpenRead("inputFilename"))
    {
        // create gzip stream
        using (var gz = new GZipStream(pcStream, CompressionMode.Compress, true))
        {
            var bytesRead = 0;
            var buff = new byte[65536]; // 64K buffer
            while ((bytesRead = inputFile.Read(buff, 0, buff.Length)) != 0)
            {
                gz.Write(buff, 0, bytesRead);
            }
        }
    }
    // The entire file has been compressed and copied to the buffer.
    // Mark the stream as "input complete".
    pcStream.CompleteAdding();

    // wait for the upload thread to complete.
    uploadThread.Join();

    // It's very important that you don't close the pcStream before
    // the uploader is done!
}

The upload thread should be pretty simple:

void UploadThreadProc(object state)
{
    var pcStream = (ProducerConsumerStream)state;
    Uploader.Upload(pcStream);
}

You could, of course, put the producer on a background thread and have the upload be done on the main thread. Or have them both on background threads. I'm not familiar with the semantics of your uploader, so I'll leave that decision to you.

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351