I am trying to download from a server a large file, about 500 Mb, but instead of saving that file to the filesystem, I am trying to process it "on the fly", retrieving some chunks of data, analysing them and, when there is enough information, saving them to the database. Here is what I am trying to do:
byte[] buffer = new byte[64 * 1024];
using (HttpResponseMessage response = await httpClient.GetAsync(Server + file, HttpCompletionOption.ResponseHeadersRead))
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
int wereRead;
do
{
wereRead = await streamToReadFrom.ReadAsync(buffer, 0, buffer.Length);
// Do the processing and saving
} while (wereRead == buffer.Length);
I tried to use the buffer of 64k as the chunks of data I need to process are about that size. My reasoning was that since I am 'awaiting' on ReadAsync, the method call will not return until the buffer is full but that is not the case. The method was returning with only 7k to 14k bytes read. I tried to use a much smaller buffer, but anyway the speed of my processing is much higher than the speed of the download, so with a 4k buffer I might have a full buffer on the first iteration but only, say, 3k on the second iteration.
Is there an approach that would be recommended in my situation? Basically, I want ReadAsync to only return once the buffer is full, or once the end of the stream is reached.
