Releasing threads during async tasks

Question

I have a system that spawns a LOT of sub-processes that must run in parallel

• The main thread for a request will spawn sub-processes and wait for them to complete.
  • Those sub-processes do some processing
  • then talk to a remote API
  • then do more processing of the results from the API
• then the main thread continues when all sub-processes are complete (or timeout hit)

We have trouble with thread counts, so we want to reduce the number of threads active by trying to release the threads whilst we wait for the remote API.

Originally we used WebRequest.GetResponse() for the API call which naturally is holding an idle thread whilst waiting for the API.

The we started using an EAP model (Event-based Async Programming ... all the various .NET methods that use IAsyncResult) , where we call BeginGetResponse(CallbackTrigger), with WaitHandles passed back to the main thread which then triggered the post-API processing.

As we understand it, this means that the sub-process thread terminates, and the Callback is triggered by a network-card-level interupt which triggers a new thread to initiate the callback. i.e. there's no thread sitting waiting to run CallbackTrigger whilst we wait for the API call.

If people could confirm this understanding that would be good?

We are now considering moving to a TPL model (Task Parallel Library ... Task<T>), using WebRequest.GetResponseAsync() which is awaitable. I'm under the impression that this is part of what await\ async does... that await passes control back up the call stack whilst the remote source waits, and that if I initiate a bunch of awaitable Tasks and then call Tasks.WaitAll then that won't be holding onto a thread for each Task whilst that task is awaiting on the remote API.

Have I correctly understood this?

Answer 1

If people could confirm this understanding that would be good?

Yes. Note that the IAsyncResult/Begin*/End* pattern is APM, not EAP. EAP would be WebClient's approach where the DownloadAsync method triggers a DownloadCompleted event when it's done.

APM/EAP are hard ways of doing asynchronous work, but are in fact asynchronous (meaning, they do not take up a thread just to block on I/O completing). They're "hard" because they makes your code much more complex - to the point that most developers never used them and just stuck with synchronous code instead.

Have I correctly understood this?

Yes. In general, all asynchronous I/O in .NET is implemented using a single I/O completion port which exists as part of the thread pool. This is true whether the API is APM, EAP, or TAP.

The whole idea of async/await with TAP is that the core Tasks (like those returned from GetResponseAsync) are still built on the same asynchronous I/O system, and then async/await makes consuming them much more pleasant; you can stay in the same method with await instead of messing with callbacks (APM) or event handlers (EAP).

As an interesting side note, Task actually implements IAsyncResult, and from a high-level perspective APM and TAP are very similar (both IAsyncResult and Task represent an operation "in flight").

You should find your TAP code significantly simpler (and easier to maintain!) than your current APM/EAP code, with no noticeable change in performance.

(On a side note, consider moving to HttpClient, which was designed from the ground up with TAP in mind, rather than HttpWebRequest/WebClient, which have had TAP bolted-on to them).

However...

I have a system that spawns a LOT of sub-processes that must run in parallel...

With this kind of a "pipeline", you may want to consider converting to TPL Dataflow. Dataflow understands both synchronous and asynchronous (TAP) work, and has built-in support for throttling. A Dataflow approach may simplify your code even further than TAP on its own.

Answer 2

Further to @Stephen Cleary's answer I set up a brief test to further prove the point.

The below code, when running the Synchronous method, with no modification to SetMinThreads, and when targetting a website that takes a few seconds to return, will hold a thread open for each request. It will show an increasing number of threads active, it start the first few Tasks instantaneously but then "choke up" as it reaches ThreadPool's limit and is only permitted to start new threads once every half second, or when an old request ends.

Setting a higher MinThreadCount postpones the problem as expected.

Keeping the MinThread Count unset, but switching to either the Asynchronous (APM) method or the Await (TAP) method causes all Tasks to be started immediately, and for the number of threads active at any point to stay low.

using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading;
using System.Threading.Tasks;

namespace LockTraceParser
{
  internal class AsyncThreadsTester
  {
    public void Run()
    {
      //ThreadPool.SetMinThreads(100, 100);

      Console.WriteLine("Beginning Test: ");
      LogThreadCounts();

      Test();
    }

    private void Test()
    {
      LogThreadCounts();

      for (int i = 0; i < 65; i++)
      {
        //StartParallelUserWorkItem(i);
        StartTask(i);
        Thread.Sleep(100); //sleep a while so that the other thread is working
        LogThreadCounts();
      }

      for (int i = 0; i < 40; i++)
      {
        Thread.Sleep(1100); //sleep a while so that the other thread is working
        LogThreadCounts();
      }
    }

    private void StartTask(int label)
    {
      var taskLabel = "Task " + label;
      Console.WriteLine("Enqueue " + taskLabel);
      Task.Run(() => GetResponseAwait(taskLabel));
    }

    private static void LogThreadCounts()
    {
      int worker;
      int io;
      ThreadPool.GetAvailableThreads(out worker, out io);
      Console.WriteLine("Worker Threads Available:" + '\t' + worker + '\t' + "IO Threads Available:" + '\t' + io + '\t' +
                        "Threads held by Process: " + '\t' + Process.GetCurrentProcess().Threads.Count);
    }


    private void GetResponseSync(object label)
    {
      Console.WriteLine("Start Sync     " + label);
      try
      {
        var req = GetRequest();
        using (var resp = req.GetResponse())
        {
          Console.WriteLine(resp.ContentLength);
        }
      }
      catch (Exception e)
      {
        Console.WriteLine("Error response " + label);
      }
      Console.WriteLine("End response   " + label);
    }

    private void BeginResponseAsync(object label)
    {
      Console.WriteLine("Start Async     " + label);
      try
      {
        var req = GetRequest();
        req.BeginGetResponse(EndGetResponseAsync, req);
      }
      catch (Exception e)
      {
        Console.WriteLine("Error Async " + label);
      }
    }

    private void EndGetResponseAsync(IAsyncResult result)
    {
      Console.WriteLine("Respond Async   ");
      var req = (WebRequest)result.AsyncState;

      using (var resp = req.EndGetResponse(result))
      {
        Console.WriteLine(resp.ContentLength);
      }
      Console.WriteLine("End Async   ");
    }

    private async Task GetResponseAwait(object label)
    {
      Console.WriteLine("Start Await     " + label);
      try
      {
        var req = GetRequest();
        using (var resp = await req.GetResponseAsync())
        {
          Console.WriteLine(resp.ContentLength);
        }
      }
      catch (Exception e)
      {
        Console.WriteLine("Error Await " + label);
      }
      Console.WriteLine("End Await   " + label);
    }

    private WebRequest GetRequest()
    {
      var req = WebRequest.Create("http://aslowwebsite.com");
      req.Timeout = (int)TimeSpan.FromSeconds(60).TotalMilliseconds;

      return req;
    }
  }
}