1

TL;DR; Give the worker process more threads!

Background

We have an ASP.NET application that expose a Web API where one of the "endpoints/methods" makes a set of queries to external system (SQL, AD etc.) and takes approx. 4 seconds.

The IO calls are synchronous so the request thread is blocked until everything is finished.

The application is written i .NET 4.6.2 and running on IIS 7.5

Problem

We experience very spiky load with 100 of request to the endpoint for a short period of time which seems to result in "thread starvation" with request queueing up as a result. This leads to unacceptable response times for the rest of the application.

CPU and memory is fine on the server during the load burst period. SQL is fine and AD is fine as well (i.e. they respond to other requests from other applications without problem)

Possible solution

My goal was to give the workerprocess more threads but cannot seem to find the correct configuration.

I've tried the recommendation in the following article: https://support.microsoft.com/en-us/help/821268/contention-poor-performance-and-deadlocks-when-you-make-calls-to-web-s

With the resulting machine.config:

<processModel autoConfig="false" maxWorkerThreads="200" maxIoThreads="200" minWorkerThreads="100"/>
<httpRuntime minFreeThreads="704" minLocalRequestFreeThreads="608"/>

but it seems to give small or no result compare to using default values.

Am i missing something?

Masus
  • 11
  • 1
  • 1
  • 2
  • Modify your ASP.NET application to use the async support, such as https://docs.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45 That's more important than changing machine.config. You also need to review source code of the web app to identify other possible threading issues. – Lex Li Mar 06 '20 at 02:37
  • Hi, I would love to do that. I cannot do that extensive changes at the moment however! Do you know of any other ways of actually increasing the number of threads that are allowed? I just cannot seems to increase the throughput. – Masus Mar 30 '20 at 14:08
  • No typical threading issues almost all require code change to remedy. There is no much room to discuss further. – Lex Li Mar 30 '20 at 16:16
  • Ok! Wierd that I cannot increase the available thread in any other way, but I take your word on it. Out of curiosity, how did you previously manage to handle larger load when async programming was not that widespread and many of the available API:s in .NET was not written using any async programming model? A couple of time consuming SQL queries and the whole request pipeline is blocked? – Masus Mar 30 '20 at 18:52
  • Due to various reasons, ASP.NET was not initially designed to be high performance. That's why ASP.NET Core has been redesigned from the scratch, and has been proven to be much better an option for new projects. I rather not comment further, as that's lengthy. There are articles over the internet on such topics if you want to explore further. – Lex Li Mar 30 '20 at 20:52

1 Answers1

2

We were facing similar issues and we resolved the same by increasing the number of worker processes instead of threads. We had 5 web applications, all hosting ASP.NET REST APIs and each of them were assigned individual AppPool. However, still we were facing spikes which were generated from the slow database/IO calls running on a single AppPool.

We then started searching around how to increase worker processes and if that would help relieve the request choking. We found this article helpful. https://medium.com/@sanjay.rajak/does-api-response-fast-if-maximum-worker-processes-is-more-than-1-4b877af6b951

We did not change anything else, no change in CPU or memory setting for the AppPools, but still the application performance was much better with very low queueing.

Virtual core count in VM = 6 Maximum Worker Processes = 3

Check if this works.

Rajarshi Basu Roy
  • 21
  • 2
  • Excellent answer and thanks for your feedback! We worked around the issue by asking the integration clients (all integration was made within the enterprise) to spread out the calls to the "heavy" endpoints. Will absolute look into your suggestion! – Masus Apr 06 '21 at 09:33
  • Cannot seem to set your answer as the answer for my question (too low reputation?) – Masus Apr 06 '21 at 09:35