How to convert Dictionary> to Task>

Question

I am working on retrieving about 10 different types of data using http-requests with rather complex dependencies between them. And I am trying to find my way through this in an elegant, readable and maintainable way without waiting unnecessarily.

Let's assume, some method creates a Dictionary<string, Task<int>>. What is the most elegant way to convert this into Task<Dictionary<string, int>>?

The new outer Task should finish, as soon as all Tasks contained in the dictionary are finished.

Of course, I can write this manually:

Dictionary<string, Task<int>> values = GetValues();
Task<Dictionary<string, int>> result = Task.Run(async () => {
    Dictionary<string, int> rewrapped = new();
    foreach (var entry in values) {
        rewrapped.Add(entry.Key, await entry.Value);
    }
    return rewrapped;
});

But isn't there a better way?

Answer 1

Rule of thumb: Never use Task.Run instead of it being truly async, it will use a thread to mimic asyncronity!

You just have to await the values, then get the Results:

Dictionary<string, Task<int>> values = GetValues();
await Task.WhenAll(values.Values);
Dictionary<string, int> results = values.ToDictionary(p => p.Key, p => p.Value.Result);

You can wrap the last two lines in a method, if you really want a Task<Dictionary<string, int>>.

async Task<Dictionary<string, int>> Unwrap(Dictionary<string, Task<int>> values)
{
    await Task.WhenAll(values.Values);
    return values.ToDictionary(p => p.Key, p => p.Value.Result);
}

Answer 2

The request is unusual. A Task isn't a value, it's a promise that something will complete in the future. To get the desired result the code will have to await all the tasks, retrieve their results, then return all of them in a Dictionary<string, int>.

There's almost certainly a better way to solve the actual problem.

One quick and dirty example would be :

async Task<ConcurrentDictionary<string,T>> GetValues<T>(CancellationToken token=default)
{
    var dict=new ConcurrentDictionary<string,T>();
    try
    {
        await Parallel.ForEachAsync(_urls,token, async (url,tk)=>{
            var res=await _httpClient.GetStringAsync(url,tk);
            dict[url]=someResult;
        });
    }
    catch(OperationCancelledException){}
    return dict;    
}

There are far better ways to solve the actual problem though - execute interdependent HttpClient requests. .NET offers several ways to construct asynchronous processing pipelines: Dataflow blocks, Channels, IAsyncEnumerable.

Dataflow Blocks

For example, using Dataflow blocks you can create a pipeline that downloads CSV files, parses them, then inserts the data into a database.

These options specify that 8 CSV files will be downloaded concurrently and two parsed concurrently.

var downloadDOP=8;
var parseDOP=2;
var tableName="SomeTable";

var linkOptions=new DataflowLinkOptions { PropagateCompletion = true};

var downloadOptions =new ExecutionDataflowBlockOptions {
    MaxDegreeOfParallelism = downloadDOP,
};

var parseOptions =new ExecutionDataflowBlockOptions {
    MaxDegreeOfParallelism = parseDOP,
};

The following code creates the pipeline

HttpClient http=new HttpClient(...);


var downloader=new TransformBlock<(Uri,string),FileInfo>(async (uri,path)=>{
    var file=new FileInfo(path);
    using var stream =await httpClient.GetStreamAsync(uri);
    using var fileStream=file.Create();
    await stream.CopyToAsync(stream);
    return file;
},downloadOptions);

var parser=new TransformBlock<FileInfo,Foo[]>(async file=>{
    using var reader = file.OpenText();
    using var csv = new CsvReader(reader, CultureInfo.InvariantCulture);
    var records = csv.GetRecords<Foo>().ToList();
    return records;
},parseOptions);

var importer=new ActionBlock<Foo[]>(async recs=>{
    using var bcp=new SqlBulkCopy(connectionString, SqlBulkCopyOptions.TableLock);
    bcp.DestinationTableName=tableName;

    //Map columns if needed
    ...
    using var reader=ObjectReader.Create(recs);
    await bcp.WriteToServerAsync(reader);
});

downloader.LinkTo(parser,linkOptions);
parser.LinkTo(importer,linkOptions);

Once you have the pipeline, you can start posting URLs to it and await for the entire pipeline to complete:

IEnumerable<(Uri,string)> filesToDownload = ...


foreach(var pair in filesToDownload)
{
    await downloader.SendAsync(pair);
}

downloader.Complete();

await importer.Completion;