6

We have an application we have recent ported to use the Aspnet boilerplate framework and are having a number of issues regarding memory usage. Our initial symptom was intense memory usage with no alleviation in times of high use. Memory use increased over night and over the weekends, but not at the rate it did during peak times. As I type this, I'm monitoring and seeing the app use upwards of 5Gb of memory on the system when a normal usage would be at most 500 - 550mb. We profiled the app using a couple of memory profiling tools and found a couple of potential leaks in framework libraries and in the version of Kestrel we were referencing, but even after repairing the issues, the footprint continued to be heavy. The behavior we see now is the app's memory usage will grow indefinitely but forcing garbage collection in the memory profiling tools will recover large swathes of it. Monitoring the app over the weekend showed that the app was operating normally under light use, but today (Monday) during peak times the app is bleeding memory again. I'm not sure what direction to go in, or how to get visibility on what the actual issue is. The memory profiling tools don't show any obvious leaks or issues in that regard, and the fact that the memory can be reclaimed by forcing garbage collection seems suspicious to me.

Francisco Garcia
  • 974
  • 1
  • 15
  • 26
  • 2
    perhaps the scheduler is just too busy to prioritise garbage collection, if you have the memory available what's more important, pausing execution to reclaim memory or servicing the next request? How much free memory do you have? – Jodrell Mar 20 '17 at 15:18
  • Right now we are going on that assumption, yes. Early last week the app was consuming memory and getting to a point where things would crash and IIS required a restart, but since we've fixed the obvious memory leaks we haven't had a crash (though we are proactively restarting the site at specific intervals). Today we are more monitoring than interfering to see if the site will run out of resources or if it is just too busy to care until it's critical. Is there any way or do you have any idea how I can prove that this is the behavior and not another undiagnosed memory leak? – Francisco Garcia Mar 20 '17 at 15:25
  • Throwing it out there, but are you using workstation (default) or server GC mode? – hythlodayr Mar 20 '17 at 15:32
  • Default. Would a change to server improve memory usage? I always thought that was for performance vs memory consumption. At this point i'll try anything twice. – Francisco Garcia Mar 20 '17 at 15:41
  • I won't claim to be an expert here, but the default GC is for UI performance. Because you don't want users' applications to freeze, the throughput of GC is less than it can be. – hythlodayr Mar 20 '17 at 15:46

2 Answers2

2

Kestrel handles garbage collection (GC) slightly different from ASPNET on IIS. Kestrel has the ability to be much faster than ASPNET and one of the ways it achieves this feat is by reducing GC pressure. Here is an article giving more detail:

https://www.poppastring.com/blog/ASPNETCoreKestrelTheNeedForSpeed.aspx

You may need to rewrite your code taking into consideration what Kestrel GC considers a long-lived object. Without any specifics I can't help much more, but it does sound like your objects are being released by your code and not being collected in a timely manner by the normal GC process.

Larry Dukek
  • 2,179
  • 15
  • 16
  • Deleted my answer. This looks far more relevant. But including the MSDN link in case it helps : https://msdn.microsoft.com/en-us/library/ee787088(v=vs.110).aspx – hythlodayr Mar 20 '17 at 16:09
  • It looks like this might be the case, but I'm not sure how to bend kestrel to my will, or at least to be a bit more reasonable with it's timings. The machine hosting the site got to 98% of memory used before the app took notice and collected enough resources to drop down to 92%.So the app is taking care of resources in perceived critical times, but i feel there's got to be something I can configure to give this behavior a bit more priority. In memory profiling, it doesn't look like there is a large amount of memory in gen2 or the loh, most of it is in gen0 and the lion's share is in unmanaged. – Francisco Garcia Mar 20 '17 at 17:31
2

Unfortunately it looks like the actual issue was buried in the framework we were using, specifically an issue with the dependency injection library not handling transient dependencies correctly. We worked with the third party provider for the library and they included a fix in a more current build, which fixed our issue 100%.

Francisco Garcia
  • 974
  • 1
  • 15
  • 26
  • Which dependency injection library were you using and what version? – David W Aug 18 '17 at 01:00
  • 1
    It was Castle Windsor, don't recall which version right now but the mismatch was due to the project being a dot net core project instead of 4.6.1, which I don't think was supported. The framework were using has a custom shim to get castle and core to play nice. The bug was in that bit of code, and has since been fixed by the developers of that framework. – Francisco Garcia Aug 18 '17 at 01:04