1

In periods we are experiencing many "Request timed out" exceptions (System.Web.HttpException) from a specific endpoint that is called often.

It appears not to be related to high-peak periods and has been experienced right after deployment and at random times. No pattern.

The solution is not to increase the execution timeout as the requests are normally completed within seconds.

Neither the web server nor the backend SQL Server is stressed. We have even seen low CPU usage during an incident period.

From ApplicationInsights I got the exact endpoint failing, which is a standard controller action. However, there is no additional information. No stack trace. No error code. Nothing. The exception is thrown at any time between 1 second and minutes after the request start.

From ApplicationInsight I can see that some of the requests to the failing endpoint are completed. However, the response time is extremely long (up to 8 minutes).

I have found nothing in the IIS logs. We have set up the failed request logging and waiting for the next incident. However, we do not expect to get more information than we already got from ApplicationInsights.

I'm uncertain whether this is an ASP.NET MVC application issue or an IIS configuration. It puzzles me, that no stack trace is available.

Any suggestions on how to approach this challenge? Pointers to articles/blogs that can help me solve the issue are very much appreciated.

UPDATE

I was looking through our trace logs and realized that they were not complete, i.e., entries were missing. We use ApplicationInsights (AI) for tracing. AI is configured to keep all traces, exceptions, and events, and it is working flawlessly in DEV and STAGING.

We have two AI environments: AI-PROD and AI-TEST. The environment is selected in web.config via instrumentation key. The entire AI config is in the ApplicationInsights.config and this file is the same in DEV, STAGING, and PROD.

I tried to connect STAGING to the AI-PROD environment to verify that it was not a problem with the environment. It worked flawlessly.

I disabled AI in PROD and the server started without throwing “Request timed out” errors during startup. When PROD is connected to either the AI-PROD or the AI-TEST environment I get “Request timed out” errors during startup.

halvorsen
  • 411
  • 1
  • 6
  • 14
  • The steps to capture process dump can be found in https://learn.microsoft.com/en-us/archive/blogs/asiatech/how-to-troubleshoot-httpexception-request-timed-out-asp-net-2-0-32-bit (you need to adapt them to the actual .NET Framework version and bitness you use). With those dumps you can analyze further. If the steps sound too complicated for you, hire an experienced consultant, or buy support services from Microsoft. – Lex Li Oct 14 '21 at 14:50
  • You can specify the request timeout (in seconds) in the web.config, like this: ` ` – Bruce Zhang Oct 15 '21 at 06:43
  • I wrote "The solution is not to increase the execution timeout as the requests are normally completed within seconds." ;-) – halvorsen Oct 15 '21 at 07:51
  • The default executionTimeout is 110s. But you said the request can complete in seconds. So the problem is actually the application. Either the application disconnects from the server before the request is processed, or the client actively disconnects. You can try to switch other browsers to see if there is a timeout problem. If so, you have to check the exception record in the event viewer. – Bruce Zhang Oct 20 '21 at 06:31
  • I'm still not certain where the problem is as I have so little information. However, it looks to me that something in the application is preventing the application from processing requests and then I get the exception. The endpoint is usually not called from a browser and it is a period production problem only. I have currently no way to to reproduce. I have already examined ApplicationInsights, standard IIS log, HTTPERR log, and the Windows Event log without getting any additional information. – halvorsen Oct 21 '21 at 09:05

0 Answers0