Run HttpClient POST in parallel

Question

I am using .net HTTPClient to save some data using POST REST API. The payload size for this API is around 10 MB. I am splitting my data in chunks and calling this POST API for each chunk. I have question mostly around approach:

I am planning to create single static instance of HTTPClient and will use same instance across application. What should be my approach? (create singleton or new client per chunk POST API call)

I would like to call all these chunk POST calls in parallel (using TASKS in .net). Is there any way to stop remaining tasks if any one task fails. I am looking for some sample code.

_factory = new TaskFactory();
_factory.StartNew(() =>
            //Call to async POST API using HttpClient                   
         ).ContinueWith((response) =>
               {
                   if (!response.IsFaulted)
                   {
                      //Do something
                   }
                   else {
                       this._logger.Error("log the error");
                   }
               });

Maybe a boolean variable that you set to true/false. The other tasks can then check that variable and stop if necessary. — Tesseract, Dec 07 '16 at 17:46
You can also do it with a cancellation token. Here is some example code https://msdn.microsoft.com/en-us/library/dd321955(v=vs.110).aspx, Also HTTPClient has a method for cancelling all pending requests. https://msdn.microsoft.com/en-us/library/system.net.http.httpclient.cancelpendingrequests(v=vs.118).aspx — Tesseract, Dec 07 '16 at 18:00
@Abhay I know this is an old post. But were you able to get this resolved? What did you finally use? — Ditty, Aug 10 '22 at 15:54

score 1 · Answer 1 · answered Dec 08 '16 at 15:36

If your calls are all to the same host, use a shared HttpClient instance.

There is no need to explicitly Create tasks using TaskFactory.StartNew in order to do I/O-bound asynchronous work in parallel. I would suggest using Task.WhenAll. Something like this:

try {
    await Task.WhenAll(chunks.Select(MakeCall));
}
catch (Exception) {
    _client.CancelPendingRequests();
}


private async Task MakeCall(string chunk) {
    var response = await _client.PostAsync(chunk);
    if (!response.IsFaulted) {
        //Do something
    }
    else {
        this._logger.Error("log the error");
        throw new Exception("call failed!");
    }
}

Run HttpClient POST in parallel

1 Answers1