Trying to resolve scalability issues by gating simultaneous requests

Question

I have a legacy application that needs to deal with scalability issues. This is a WCF service that listens to requests received from a back end system and does some computations/processing of data based on those requests. The computations are not cpu intensive although in some cases it calls some third party library APIs (which itself calls SOAP based web-services). There aren't any database calls involved. Here is how the service is been set up.

[ServiceContract]
public interface IMyService
{
    [OperationContract]
    ProcessingResult ProcessData(int dataId, string dataDescription);
}


[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple, IncludeExceptionDetailInFaults = true)]
public class MyService : IMyService
{

    ProcessingResult ProcessData(int dataId, string dataDescription)
    {
         // Some data processing using third party library APIs and return ProcessingResult instance.

    }
}

public class ProcessingResult
{
    public int Code;
    public string Message;
}

The issue here is that under load conditions, there are so many calls of ProcessData method that overloads the system. In these scenarios,I have seen hundreds of threads running at a time. They eventually do complete the task but system slows down drastically. In order to deal with it, I am considering to add some sort of gating mechanism. Something like delegating the work to another class that queue the processing data calls to a ThreadPool with threshold on maximum number of requests process simultaneously. The problem I see with this approach is that ProcessData still need to wait for the work queued in threadpool to finish before returning ProcessingResult instance.

Googling seem to suggest that async-await could be a nice pattern, however, I am little restricted as this application isn't using latest and greatest .net versions and it could be a big ask to migrate to newer .NET version at this point. Any suggestions on how can I use ThreadPool.QueueUserWorkItem mechanims in this class with ability to wait on a particular items to finish?

Could you explain the problem a bit more? Is this a Web API that needs to respond right away? Or can the data processing happen hours later? Who calls this? Is the bottle neck the CPU, some external database, memory usage? async / await is probably not your solution ... unless the problem is running out of threads, and the threads are just waiting for external response. — hatcyl, May 27 '15 at 04:17
@hatcyl I added some more detail in the question but this is a case of high number of threads that does increase memory pressure on system — Daniel Gabriel, May 27 '15 at 10:28

score 1 · Answer 1 · answered May 27 '15 at 11:43

If you need to handle hundreds of simultaneous requests then non-blocking IO is important. It removes the need for so many threads. In .NET 3.5 there are no convenient options to do this. If you can make it to 4.0 the story becomes 10x better.

What I would do in 4.0+ is:

static SemaphoreSlim sem = new SS(100); //max 100 concurrent requests
async Task<ProcessingResult> ProcessData(...) {
 await sem.WaitOne();
 await ProcessRequestAsync(); //or, do it synchronously here if convenient
 sem.Release();
}

On pre-4.0 you need to find a non-blocking way to do the throttling that the semaphore provides. That will likely involve using IAsyncResult.

Or, you find a way in WCF to throttle the max number of concurrent requests. That would solve the problem, too.

Even if you offload to background threads this helps you little: You need to expose an IAsyncResult to WCF so that the WCF thread can be unblocked.

there are so many calls of ProcessData method that overloads the system. In these scenarios,I have seen hundreds of threads running at a time.

What is causing a problem here? Too many threads or too many concurrent operations overloading the backend? If it is the former, then a quick and totally valid fix is to increase the thread-pool limits. 1000 threads do not cause issues in my testing. If the backend is overloaded, unlimit the thread-pool as well and use a synchronous semaphore to limit the number of concurrent backend calls.

So here are some alternatives to pick from. The best one would be async IO and waiting but that is harder to pull off.

score 0 · Answer 2 · answered May 27 '15 at 16:40

If it really is just to many threads waiting for IO, I can think of two solutions:

async / await WILL help. It will keep the number of threads down. You should be able to get the libraries: NuGet async / await

or

You can simply get another server, a load balancer, and have your requests split between the two servers.

Trying to resolve scalability issues by gating simultaneous requests

2 Answers2