How do you compress a stream and upload it to Azure Blob Storage without writing to a file?

Question

I have written a Post method in ASP.NET Core to compress the requests body and upload it to Azure Blob Storage. The method takes parameters as follows:

public async Task<IActionResult> Post([FromHeader] string AssignmentId)

Various strings are then set, including fetching the connection string for the storage:

string fileName = $"{AssignmentId}.gz";
string compressedFilePath = Path.Combine(hostEnvironment.ContentRootPath, $"Test JSONs/{fileName}");
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");

I initialize the BlobClient:

BlobClient blobClient = new BlobClient(connectionString, "assignments", fileName);

Then I create a file, and compress the body stream of the request using GZipStream to the file:

using (FileStream compressedFileStream = System.IO.File.Create(compressedFilePath))
{
    using GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress);
    using Stream bodyStream = HttpContext.Request.Body;
    await bodyStream.CopyToAsync(compressionStream);
}

Finally I read the file I just wrote and upload using the FileStream:

using (FileStream fileStream = System.IO.File.OpenRead(compressedFilePath))
{
    await blobClient.UploadAsync(fileStream);
}

This solution works, but I am concerned about the constant reading and writing of the file, in terms of speed. I attempted to use a MemoryStream passed into the GZipStream, however it ended up only uploading 10B files when the files should be 1KB+.

I appreciate any suggestions.

Here is the complete method:

public async Task<IActionResult> Post([FromHeader] string AssignmentId)
{
    string fileName = $"{AssignmentId}.gz";
    string compressedFilePath = Path.Combine(hostEnvironment.ContentRootPath, $"Test JSONs/{fileName}");
    string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");

    BlobClient blobClient = new BlobClient(connectionString, "assignments", fileName);
    
    using (FileStream compressedFileStream = System.IO.File.Create(compressedFilePath))
    {
        using GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress);
        using Stream bodyStream = HttpContext.Request.Body;
        await bodyStream.CopyToAsync(compressionStream);
    }

    using (FileStream fileStream = System.IO.File.OpenRead(compressedFilePath))
    {
        await blobClient.UploadAsync(fileStream);
    }
    
    return Ok();
}

aside from the BLOB storage upload, it's really a duplicate of this: https://stackoverflow.com/questions/10599596/compress-and-decompress-a-stream-with-compression-deflatestream — Mitch Wheat, Mar 05 '21 at 01:14
@MitchWheat The answer to that problem uses .Close() on the compression stream, but when I use that, it also closes the MemoryStream passed into the compression stream. — Alasdair Cooper, Mar 05 '21 at 12:16
https://stackoverflow.com/questions/63623674/how-to-compress-files-to-zip-and-upload-to-azure-blob-storage — Mitch Wheat, Mar 06 '21 at 00:02

score 1 · Accepted Answer · answered Mar 10 '21 at 00:51

I eventually solved this by both leaving the compression stream open, and by resetting the position of the memory stream the compression stream is writing to (thanks to @MitchWheat !).

using MemoryStream memoryStream = new MemoryStream() ;
using (Stream bodyStream = HttpContext.Request.Body)
{
    using (GZipStream compressionStream = new GZipStream(memoryStream, 
    CompressionMode.Compress, true))
    {
        await bodyStream.CopyToAsync(compressionStream);
    }
}
memoryStream.Position = 0;

await blobClient.UploadAsync(memoryStream, overwrite: true);

score 0 · Answer 2 · answered Jun 09 '22 at 22:26

The Azure Storage library offers a push-based stream to upload content that works well with compression.

const int PageSize = 1024 * 1024; // 1 MB
using var sourceFile = new FileStream(file.FullName, FileMode.Open, FileAccess.Read, FileShare.Read);
using var destinationStream = await blobClient.OpenWriteAsync(overwrite: true, new() { BufferSize = PageSize }, cancellationToken);
using var compressedContent = new GZipStream(destinationStream, CompressionMode.Compress, leaveOpen: true);
await sourceFile.CopyToAsync(compressedContent, cancellationToken);

How do you compress a stream and upload it to Azure Blob Storage without writing to a file?

2 Answers2