C# how to aggregate results from multiple HTTP requests?

Question

I am designing an application written in C# and AspNetCore that requires making multiple HTTP requests to microservices and aggregating the results before returning to the controller.

I have looked into MediatR pipelines but seems limited and unfit for this particular scenario.

I imagine there being something useful within the realm of map and reduce but struggling to find a framework my developers can get going with quickly.

We want to ensure aggregation is terminated should one or more requests fail.

I created a proof of concept using worker threads e.g.

var results = await Task.WhenAll(task1, task2);

But this does not provide a framework like MediatR developers can easily adopt and unit test.

I suppose the same can be said had my proof of concept been written using Rx extensions and Observable.zip. I am nonetheless rather curious and hoping to find examples of using Observable.zip in C# similar to what has been demonstrated in this Netflix blog post

Reactive Programming in the Netflix API with RxJava

Rx extensions for .Net core are available from System.Reactive namespace — sramalingam24, Mar 08 '18 at 14:37
Rx is extremely powerful. There are many operators that can be combined in many ways. Without knowing precisely the source data, the signatures of your HTTP requests, and how you want to combine the results it is really impossible to give you a meaningful answer. Could you please provide a [mcve] of how you would call your requests and what the final form your data needs to take? — Enigmativity, Mar 08 '18 at 22:17

score 0 · Answer 1 · answered Mar 20 '18 at 15:36

I have built an example to help me understand the capabilities in using Rx in .NET.

Here is the code.

class Program
{
    static void Main(string[] args)
    {
        new Program().Run().Wait();
    }

    public async Task Run()
    {


        var resources =
            new List<string>
                {
                    "http://www.google.com"
                    ,"http://www.yahoo.com"
                    ,"http://cnn.com"
                    ,"http://microsoft.com"
                }
                .ToObservable()
                .Select(s => Observable.FromAsync(() => DownloadAsync(s)))
                .Concat()
                .Where(w => w.IsSuccessStatusCode)
                .Select(s => Observable.FromAsync(() => GetResources(s)))
                .Concat()
                .Where(w => w.Any())
                .SelectMany(s => s)
                .ToEnumerable()
                .OrderBy(o => o.Name);

        foreach (var re in resources)
        {
            Console.WriteLine(re.Name);
        }

        Console.ReadLine();
    }

    public async Task<HttpResponseMessage> DownloadAsync(string url)
    {
        var request = new HttpRequestMessage(HttpMethod.Get, url);

        var client = new HttpClient();

        return await client.SendAsync(request);
    }

    public async Task<IEnumerable<Resource>> GetResources(HttpResponseMessage message)
    {
        var ignoreContent = await message.Content.ReadAsStringAsync();
        return new List<Resource> { new Resource { Name = message.Headers.Date.ToString() } };
    }
}
public class Resource
{
    public string Name { get; set; }
}

C# how to aggregate results from multiple HTTP requests?

1 Answers1