Why is Stream.CopyTo not writing directly to a file?

Question

This action block is connected to a transform block with the signaturevar getStream = new TransformBlock<FileChunk, Tuple<Task<HttpResponseMessage>, FileChunk>> however, the stream is not writing directly to the file provided. For some reason, I think that it is still being cached to memory. The Task is the tuple is initalized as client.SendAsync(request, HttpCompletionOption.ResponseContentRead, CancellationToken.None);

var writeStream = new ActionBlock<Tuple<Task<HttpResponseMessage>, FileChunk>>(async task =>
{
    using (var streamToRead = await task.Item1.Result.Content.ReadAsStreamAsync())
    {
        using (var fileToWriteTo = File.Open(task.Item2._tempfilename,
            FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite))
        {
            await task.Item1.Result.Content.CopyToAsync(fileToWriteTo).ContinueWith(task1 =>
            {
                var s = new FileChunk();
                Interlocked.Add(ref TasksDone, 1);
                asyncTasks.TryDequeue(out s);
            }, CancellationToken.None, TaskContinuationOptions.OnlyOnRanToCompletion,
                TaskScheduler.Current);
        }
    }
}, new ExecutionDataflowBlockOptions
{
    BoundedCapacity = Environment.ProcessorCount, // Cap the item count
    MaxDegreeOfParallelism = Environment.ProcessorCount, // Parallelize on all cores
});

Any advice on how I can fix this? Also, I am assuming that it is caching to memory first because the file size does not update or increase on refresh, rather all at once.

What's going on with the task continuation? Particularly the `out s` - that's not going anywhere. — Enigmativity, Jul 15 '20 at 04:59
It's also a little weird that you've got a `ConcurrentQueue` called `asyncTasks`. — Enigmativity, Jul 15 '20 at 05:01
@Enigmativity it's not really supposed to go anywhere. I'm using a custom ConcurrentQueue for calling events on Enqueue and TryDequeue. You can see exactly what I'm using this for on my [github repo](https://github.com/gregyjames/OctaneDownloader) — 0xSingularity, Jul 15 '20 at 06:03
Ideally you should post enough detail in your question to be able to see what's going on. One thing I saw when I had a quick look at your code in the repo is the use of `try` `catch (Exception ex) { .. }` code throughout. That's just great way of swallowing errors. Who knows if that's part of this problem. You should remove it all and only catch specific exceptions that you can meaningfully handle. — Enigmativity, Jul 15 '20 at 06:35
Try change your HttpCompletionOption on SendAsync to`HttpCompletionOption.ResponseHeadersRead`. This will allow you to stream the response content directly from the socket as soon as the headers are available, rather than buffering the response to memory before you can access it. — SeanOB, Jul 31 '20 at 00:27

score -1 · Answer 1 · answered Jul 15 '20 at 06:50

I suggest to detach the artificial ContinueWith continuation, and await directly the CopyToAsync operation. With your current setup you'll get no information if something goes wring, because in case of an exception the TaskContinuationOptions.OnlyOnRanToCompletion will cause the continuation to become cancelled, and the TPL Dataflow blocks ignore OperationCanceledExceptions.

Here is my suggestion:

var writeStream = new ActionBlock<Tuple<Task<HttpResponseMessage>, FileChunk>>(async task =>
{
    using (var streamToRead = await task.Item1.Result.Content.ReadAsStreamAsync())
    {
        using (var fileToWriteTo = File.Open(task.Item2._tempfilename,
            FileMode.OpenOrCreate, FileAccess.Write, FileShare.None))
        {
            await task.Item1.Result.Content.CopyToAsync(fileToWriteTo);
        }
        Interlocked.Add(ref TasksDone, 1);
        asyncTasks.TryDequeue(out var s);
    }
});

I also suggest to consider replacing the Tuple with ValueTuple, in order to get a simplified syntax and prevent redundant allocations:

var writeStream = new ActionBlock<(Task<HttpResponseMessage>, FileChunk)>(async task =>

Finally I should note that I am skeptical about the performance characteristics of a procedure that tries to write multiple files in the same physical disk concurrently.

Why is Stream.CopyTo not writing directly to a file?

1 Answers1