I'm stuck figuring out what will be the next trouble shooting step to take or research.
I've got one web application running on one IIS8 webserver. Lately we are struggling with the performance of the webapplication. This morning the webapplication 'crasht' again. With this I mean that it didn't respond to any new requests.
Because Perfmon is your friend, I fired off the following counters:
- Processor Time (%)
- Requests Executing
- Requests/Sec
See the image I've included.
What I find interessting is that "Requests Executing" only increased at this point. And... the CPU was not running 100% to get rid of all these requests.
As I'm the main developer of this web application, I already optimized many high-CPU webpages. With this SO thread I'm hoping to find the bottle neck. Maybe with tips for extra logging. Application code analysis etc. I can provide more information about the setup or application when needed.
Hope you can help. Many thanks in advance.
Some specifications:
- Windows Server 2012R2
- IIS8
- Intel Xeon Quad core CPU
- Total of 4 gig memory
- ASP.net 4.0 web application
User specs:
- Google Analytics Real Time visitors between 100-500
- Normal requests/sec average of 30
- 'Peak' requests/sec of 200 / 300
EDIT: I run a Perfmon Data Collection Set for about a hour. Exactly at the moment when the website crashed. I used the tool PAL to analyze it. It only had many warnings about the memory being below 5%. This memory issue is clearly an issue we will be resolving soon.
Another thing I noticed is that the list of "current requests" in IIS8 was enormous. It contained hundreds of "current requests". I would expect these requests to reach a time out value and send a request Timed Out to the users.
Here is a print-screen of the current healthy situation:
This list was infinite long.
And, again another thing I just noticed, was one request taking more then 10 minutes(!) to deliver a byte[] to a user. The 'state' was SendResponse. I persume this user was on a low-bandwitdh device. As long as this user is downloading - one workerprocess is taken. How should we prepare this long pending requests?