2

I am troubleshooting a high traffic C# .NET Framework 4.61 website that utilizes System.Runtime.Caching.MemoryCache quite extensively. In the most recent week, I've seen slowdowns and when I put up some Perfmon counters, I saw that the MemoryCache gets emptied out every two minutes.

enter image description here

All other counters in the image show a similar picture (e.g. Cache Hits, Misses, etc...).

Initially, I thought that perhaps the app is running up against maximum limits of MemoryCache. So I added the following to the web.config:

<system.runtime.caching>
  <memoryCache>
    <namedCaches>
      <add name="CacheManager"
           cacheMemoryLimitMegabytes="8000"
           physicalMemoryLimitPercentage="99"
           pollingInterval="00:05:00" />
    </namedCaches>
  </memoryCache>
</system.runtime.caching>

However, nothing changed and the MemoryCache continues to be dumped every 2 minutes. Some other troubleshooting notes:

  • There is no memory pressure. The application gets up to 2-3 GB of RAM before the 2 minute mark and then memory drops a bit (as evidenced by the # Bytes in all Heaps counter). The server has 24 GB of RAM.
  • Every 2 minutes there is a CPU spike because the app has to go back to the database to fetch data again.
  • The application is running on IIS in Windows Server 2019. It was previously running on Windows Server 2008, where these issues didn't exist.

So what could cause the MemoryCache to drop every 2 minutes?

P.S. With the help of MS Support and a couple of proc dumps, we were able to determine that the application domain is restarting every 2 minutes. We've added HKLM\Software\Microsoft\ASP.NET\FCNMode=1 to disable ASP.NET from responding to file system changes. However, what actually causes the restart is still unknown. According to Procmon, no changes are happening application directory.

AngryHacker
  • 59,598
  • 102
  • 325
  • 594
  • Is it possible to use an external caching service, even if it's running on the same computer, such as Redis? – gunr2171 Nov 15 '21 at 19:04
  • @gunr2171 There is nothing else running on the computer other than IIS. The app does use Memcached external caching service as well, but very sparingly. Memcached is on a different server. – AngryHacker Nov 15 '21 at 19:12
  • What are the app pool settings in IIS? Does your worker, website, or web app turn off on idle? – gunr2171 Nov 15 '21 at 19:14
  • @gunr2171 The app pool is all defaults. So yes, idle timeout is in 20 minutes. However, the server is processing requests all the time. The website is also all defaults. The worker itself doesn't get recycled, because the PID is the same between cache drops. – AngryHacker Nov 15 '21 at 19:19
  • 2
    Stupid question, but is the app 32-bit (for example Any + prefer 32-bit). Two to three gog is exactly memory pressure in 32-bit land – Flydog57 Nov 15 '21 at 19:27
  • @Flydog57 I wish it were that simple. 32-bit apps are disabled in the app pool. I've added a postscript to the question. Turns out the app restarts every 2 minutes. But why - no idea yet. – AngryHacker Nov 16 '21 at 00:20
  • If you've got MS Support involved, looking at dumps, no one here is going to be able to give you better advice – Flydog57 Nov 16 '21 at 02:28
  • @Flydog57 From my memory of dealing MS Tech support years ago, typically you'd have to go through 2-3 people before you get to someone competent, so I wasn't expecting an answer from them soon. That's why I posted here. However, this time, the first person was super competent. – AngryHacker Nov 16 '21 at 17:23

0 Answers0