4

In order to support upload of large (actually very large, up to several gigabytes) files with progress report we started using HttpClient with PushStreamContent, as described here. It works straightforward, we copy bytes between two streams, here is a code example:

    private void PushContent(Stream src, Stream dest, int length)
    {
        const int bufferLength = 1024*1024*10;
        var buffer = new byte[bufferLength];
        var pos = 0;
        while (pos < length)
        {
            var bytes = Math.Min(bufferLength, length - pos);
            src.Read(buffer, 0, bytes);
            dest.Write(buffer, 0, bytes);
            pos += bufferLength;
            dest.Flush();
            Console.WriteLine($"Transferred {pos} bytes");
        }
        dest.Close();
    }

But in the beginning this code raised OutOfMemory exception after transferring 320 MB even though the memory consumption of the process was not very high (ca. 500 MB). What fixed this problem is setting TransferEncodingChunked flag:

request.Headers.TransferEncodingChunked = true;

Not only we were able to transfer huge files with this flag set, the memory consumption decreased by 90%.

I haven't found any documentation requiring use of TransferEncodingChunked, it was more of a trial and failure process but it seems crucial in this scenario. Still I am puzzled why the exception is thrown at all - the memory consumption is not very high, what causes it?

Vagif Abilov
  • 9,835
  • 8
  • 55
  • 100

1 Answers1

4

Chunked transfer encoding

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of "chunks". It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require.1 Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content.

The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

If we think logically, the file is send on little chunks, that means when you finish with a chunk you are freeing it from the memory. In the end you have less memory consumption, because you are dealing with multiple small chunks.

mybirthname
  • 17,949
  • 3
  • 31
  • 55
  • This makes sense and explains why memory consumption goes down. Still it's strange that OutOfMemory exception is thrown at all. If I don't use PushStreamContent and stick with tranditional StreamContent fully controlled by HttpClient, no exception is raised. – Vagif Abilov Oct 11 '16 at 09:15
  • 1
    @VagifAbilov I can't tell you why, this need research and it is a different question. You can check this question: http://stackoverflow.com/questions/16168683/webapi-streamcontent-vs-pushstreamcontent. Pretty much they said that push stream content is using when you need to push data to stream, and Stream Content when you need to pull from stream. – mybirthname Oct 11 '16 at 09:24
  • Thanks for the link, quite useful. – Vagif Abilov Oct 11 '16 at 09:26