0

I've got an application that:

  • Targets C# 6
  • Targets .net 4.5.2
  • Is a Windows Forms application
  • Builds in AnyCPU Mode beacuse it...
  • Utilizes old 32 bit libraries that cannot be upgraded to 64 bit, unmanaged memory
  • Uses DevExpress, a third party control vendor
  • Processes many gigabytes of data daily to produce reports

After a few hours of use in jobs that have many plots, the application eventually runs out of memory. I've spend quite a long time cleaning up many leaks found in the code and have gotten the project to a state where at the worst case it may be using upwards 400,000K of memory at any given time, according to performance counters. Processing this data has not yielded any issues at this point since data is processed in Jagged arrays, preventing any issues with the Large Object Heap.

Last time this happened the user was using ~305,000K of memory. The application is so "out of memory" that the error dialog cannot even draw the error icon in the MessageBox that comes up, the space where the icon would usually be is all black.

So far I've done the following to clean this up:

  • Windows forms utilize the Disposed event to ensure that resources are cleaned up, dispose is called manually when required
  • Business objects utilize IDisposable to remove references
  • Verified cleanup using ANTS memory profiler and SciTech memory profiler. The low memory usage suggests this is not the case but I wanted to see if I saw anything that could be helpful, I could not
  • Utilized the GCSettings.LargeObjectHeapCompactionMode property to remove any fragmentation from processing data that may be fragmented in the Large Object Heap (LoH)

Nearly every article that I've used to get to this point suggests that out of memory actually means out of contiguous address space and given the amount that's in use, I agree with this. I'm not sure what to do at this point since from what I understand (and am probably very wrong about) is that the garbage collector clears this up to make room as the process moves along, with the exception of the LoH, which is cleaned up manually now using the new LargeObejctHeapCompactionMode property introduced in .net 4.5.1.

What am I missing here? I cannot build to 64 bit due to the old 32 bit libraries that contain proprietary algorithms that we do not have access to even dream of producing a 64 bit version of. Are there any modes in these profiles I should be using to identify exactly what is growing out of control here?

If this address space cannot be cleared up does this mean that all c# applications will eventually run "out of memory" because of this?

Mohgeroth
  • 1,617
  • 4
  • 32
  • 47
  • Any chance of "calling out" to the 32-bit libraries via an API so that you can run your application in x64? – Martin May 14 '19 at 19:23
  • 1
    Take a look in TaskManager, see how many handles, GDI and User objects your application is generating. – Jimi May 14 '19 at 19:49
  • 4
    @gilliduck: This is a perfectly reasonable "no code" question and clearly on topic; the question is about problem diagnosis techniques; the problem is clearly stated, and we know what the original poster has already tried, what they've observed, and where they're stuck. – Eric Lippert May 14 '19 at 22:26
  • 1
    @Jimi: That's good advice, but the better tool to use is sysmon, not taskman. Sysmon is a far, far more sophisticated tool for analyzing these sorts of problems. https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon – Eric Lippert May 14 '19 at 22:28
  • Btw LargeObejctHeapCompactionMode provides ability to compact the LOH only once, after this the property is set to the default mode. Are you sure that you don't have continuous problems with LOH fragmentation? – Dmytro Mukalov May 15 '19 at 07:00
  • @DmytroMukalov I make sure to set this back to CompactOnce anytime I manually collect garabage. – Mohgeroth May 15 '19 at 14:44
  • @Martin I read some articles on this that mentioned running these libraries as a service (or something to that effect) which seemed like an awful lot of overhead. After consideration I decided this doesn't actually solve the problem and just hides it in more memory space. I've worked on this for quite some time so I would have loved to move this to 64 bit while I worked on a long term solution, but the overhead seems quite much. – Mohgeroth May 15 '19 at 14:47
  • If LOH is get fragmented faster than period between manual collections elapses, it can be not enough. – Dmytro Mukalov May 15 '19 at 14:52
  • Based on input from the community above I've done some additional testing and found lots of handle activity that looks to be the culprit in this case. These things don't stand out to me in ANTS (perhaps I'm not using the right settings?) but this is very clear in the task manager, thanks for the feedback everyone! – Mohgeroth May 15 '19 at 14:54
  • @DmytroMukalov Noted, thanks for the feedback, I'll double check this as I reproduce scenarios that approach critical mass to see what's happening at that point in the app. – Mohgeroth May 15 '19 at 14:55

1 Answers1

3

Nearly every article that I've used to get to this point suggests that out of memory actually means out of contiguous address space and given the amount that's in use, I agree with this.

This is a reasonable hypothesis, but even reasonable hypotheses can be wrong. Yours probably is wrong. What should you do?

Test it with science. That is, look for evidence that falsifies your hypothesis. You want to assume that it is anything else, and be forced by the evidence you've gathered that your hypothesis is not false.

So:

  • at the point where your application runs out of memory, is it actually out of contiguous free pages of the necessary size? It sure sounds like your observations do not indicate that this is true, so the hypothesis is probably false.

What is other evidence that the hypothesis might be false?

  • "After a few hours of use in jobs that have many plots, the application eventually runs out of memory."
  • "Uses DevExpress, a third party control vendor"
  • "the error dialog cannot even draw the error icon in the MessageBox"

None of this sounds like an out of memory problem. This sounds like a third party control library leaking OS handles for graphics objects. Unfortunately, such leaks usually surface as "out of memory" errors and not "out of handles" errors.

So, that's a new hypothesis. Look for evidence for and against this hypothesis too. You're doing a good job by using a memory profiler. Use a handle profiler next.

If this address space cannot be cleared up does this mean that all c# applications will eventually run "out of memory" because of this?

Nope. The GC does a good job of cleaning up managed memory; lots of applications have no problem running forever without leaking.

Eric Lippert
  • 647,829
  • 179
  • 1,238
  • 2,067
  • 1
    I've started digging into handles and do see lots of activity when plotting, this is likely where I need to look, both on my end and on the third party side. Thankfully they respond within a day and provide fixes shortly thereafter, especially if a core component like this is leaking heavily. Thanks for clear answers to my questions, much appreciated! – Mohgeroth May 15 '19 at 14:39