I have an ASP.NET 4.0 application that has been growing over the years in terms of number of pages and number of daily users. At one point it started failing with "HTTP Error 503. The service is unavailable." once almost every day at about 12PM (peak load time). Restarting the app pool fixed it for another day or two.
Examining the system logs showed that the app pool was recycled about every minute and once in a blue moon Rapid Fail Protection kicked in and stopped the app pool. Here is a typical scenario (abbreviated for clarity):
2014-06-10 07:30:53 5117 4 Information event A worker process with process id of '4060' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:31:53 5117 4 Information event A worker process with process id of '16764' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:33:53 5117 4 Information event A worker process with process id of '18632' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:34:53 5117 4 Information event A worker process with process id of '49096' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:35:53 5117 4 Information event A worker process with process id of '8960' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:36:53 5117 4 Information event A worker process with process id of '31384' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:37:53 5117 4 Information event A worker process with process id of '37244' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:38:53 5117 4 Information event A worker process with process id of '12864' serving application pool 'slepov-27' has requested a recycle because it reached its private bytes memory limit.
2014-06-10 07:39:11 5002 1 Error event Application pool 'slepov-27' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Disabling Rapid Fail Protection was not an option as the app runs on a shared hosting.
My guess is that it was all those recycles that trigger Rapid Fail Protection and the recycles were caused by exceeding memory limits on the app pool. The memory limit is set at 400M (not sure which, Working Set or Commit Size - emailed support).
I have set on to find out the typical memory usage of my app in a one user scenario. I have run it up in IIS (7) on my dev machine and looked at Working Set and Commit Size in Task Manager. One thing that surprised me was this: the memory went up by about 3-6 megs with every page I opened while the pages themselves did not contain much code, just mark-up. Given that the web site is about 100 pages, going over 400M did not seem unlikely.
Another surprise was pages that actually did do some processing (mostly in-memory calculations, no database or web service calls). And the problem with such pages was that they ramped up memory by 200M and that memory was not reclaimed. I was sure there was nothing holding on to that memory and I confirmed that with ANTS. 'Bytes in all heaps' goes back after the page finishes processing but 'Private Bytes' does not.
To isolate the problem even further I have put together a test app (source code), two apps in fact, one console and one asp.net. All they do is let you allocate 1Gb of memory and then call GC.Collect. Very simple. But the results were strikingly different between console and asp.net:
Working Private Commit Run the console app
6 1 5 About to allocate 1Gb. Press ENTER to continue ...
1082 1076 1082 Allocated 1Gb. Press ENTER to deallocate ...
1082 1076 1082 Press ENTER to GC.Collect ...
16 10 16 Press ENTER to exit ...
Run the ASP.NET app
36 15 102 New up 1Gb
1131 1110 1326 GC.Collect
570 548 764 Wait 10 mins
570 548 764 New up 1Gb
1132 1111 1322 GC.Collect
575 554 764 GC.Collect
575 554 764
As you can see the console app (almost) goes back to its original memory usage after GC.Collect is called while the asp.net app keeps holding on to more than half a gig of memory.
As seen in this ANTS screenshot, the managed memory (Bytes in all heaps) is freed, but the unmanaged (Private Bytes) is not entirely freed. Why?
I have set <compilation debug="false" /> as advised by many sources.
I have looked at these similar questions (and many more):
To sum up, my main question is why unmanaged memory is not freed entirely after GC.Collect? Also it would be great if someone could explain the gradual stepping up in memory usage with every new open page. Thanks.
