Why does background garbage collection sometimes pause my application and how can I prevent it?

Question

We have a large (= it may need 200 - 500 MB or more memory in the course of a day) WPF application that is used for multiple hours a day. Occasionally, the application hangs for no apparent reason. Analysing process dumps shows that garbage collection is active and the reason for the paused application.

We are using .NET 4.0, and as far as I understand the newly introduced "background garbage collection" should reduce the amount of time the whole process is blocked for garbage collection (foreground garbage collection).

However, even if the pauses occur not that often, they are interrupting the workflow when they take more than a few seconds (which is the case).

This leads me to the following questions:

I read that in a server environment, background garbage collection is not used by default. Our application does in fact run on a server operating system (Windows Server 2003 R2 x64), despite being a client application (and not a server app). Does this mean background garbage collection is not used? Or does it only apply to services / ASP.NET?
Assuming that background garbage collection is indeed enabled, how can I prevent foreground collections from happening too often / taking too long? My current approach would be to detect idle periods in the application (eg. app not being used for 5 minutes or more) and initiate a forced garbage collection so that it doesn't happen at a later point of time when the application is being used.

Debug Diag process dump info: enter image description here

Stack trace:

mscorlib_ni!System.GC.Collect(Int32, System.GCCollectionMode)+47 
[[InlinedCallFrame] (System.GC._Collect)] System.GC._Collect(Int32, Int32) 
PresentationCore_ni!MS.Internal.MemoryPressure.ProcessAdd()+1d0 
PresentationCore_ni!MS.Internal.MemoryPressure.Add(Int64)+39 
PresentationCore_ni!System.Windows.Media.SafeMILHandleMemoryPressure..ctor(Int64)+43 
PresentationCore_ni!System.Windows.Media.SafeMILHandle.UpdateEstimatedSize(Int64)+38 
PresentationCore_ni!System.Windows.Media.Imaging.RenderTargetBitmap.FinalizeCreation()+df 
PresentationCore_ni!System.Windows.Media.Imaging.RenderTargetBitmap..ctor(Int32, Int32, Double, Double, System.Windows.Media.PixelFormat)+d9 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.HostUtils.GetRenderTargetBitmapForVisual(Int32, Int32, System.Windows.Media.Visual)+b1 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.HostUtils.GetBitmapForFrameworkElement(System.Windows.FrameworkElement)+89 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.HostUtils.GetBitmapForTransparentWindowsFormsHost(System.Windows.Forms.Integration.WindowsFormsHost)+4b 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.HostUtils.GetBitmapForWindowsFormsHost(System.Windows.Forms.Integration.WindowsFormsHost, System.Windows.Media.Brush)+1f 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.WindowsFormsHostPropertyMap.BackgroundPropertyTranslator(System.Object, System.String, System.Object)+109 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.PropertyMap.RunTranslator(System.Windows.Forms.Integration.PropertyTranslator, System.Object, System.String, System.Object)+32 
WindowsFormsIntegration_ni!System.Windows.Forms.Integration.WindowsFormsHost.ArrangeOverride(System.Windows.Size)+277 
PresentationFramework_ni!System.Windows.FrameworkElement.ArrangeCore(System.Windows.Rect)+8e3 
PresentationCore_ni!System.Windows.UIElement.Arrange(System.Windows.Rect)+385 
PresentationCore_ni!System.Windows.ContextLayoutManager.UpdateLayout()+2b5 
PresentationCore_ni!System.Windows.ContextLayoutManager.UpdateLayoutCallback(System.Object)+19 
PresentationCore_ni!System.Windows.Media.MediaContext+InvokeOnRenderCallback.DoWork()+10 
PresentationCore_ni!System.Windows.Media.MediaContext.FireInvokeOnRenderCallbacks()+76 
PresentationCore_ni!System.Windows.Media.MediaContext.RenderMessageHandlerCore(System.Object)+8a 
PresentationCore_ni!System.Windows.Media.MediaContext.AnimatedRenderMessageHandler(System.Object)+6e 
WindowsBase_ni!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)+53 
WindowsBase_ni!MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)+42 
WindowsBase_ni!System.Windows.Threading.DispatcherOperation.InvokeImpl()+8d 
WindowsBase_ni!System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(System.Object)+38 
mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+51 
[[HelperMethodFrame_PROTECTOBJ] (System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup)] System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object) 
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+6a 
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+7e 
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+2c 
WindowsBase_ni!System.Windows.Threading.DispatcherOperation.Invoke()+68 
WindowsBase_ni!System.Windows.Threading.Dispatcher.ProcessQueue()+15e 
WindowsBase_ni!System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)+63 
WindowsBase_ni!MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)+be 
WindowsBase_ni!MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)+7d 
WindowsBase_ni!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)+53 
WindowsBase_ni!MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)+42 
WindowsBase_ni!System.Windows.Threading.Dispatcher.InvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)+b4 
WindowsBase_ni!MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)+104 
WindowsBase_ni!DomainBoundILStubClass.IL_STUB_PInvoke(System.Windows.Interop.MSG ByRef)+3c 
[[InlinedCallFrame]] 
WindowsBase_ni!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)+c1 
WindowsBase_ni!System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame)+49 
PresentationFramework_ni!System.Windows.Application.RunDispatcher(System.Object)+5b 
PresentationFramework_ni!System.Windows.Application.RunInternal(System.Windows.Window)+74 
PresentationFramework_ni!System.Windows.Application.Run(System.Windows.Window)+2b

What evidence do you have to support your claim that *garbage collection sometimes pause my application*? It is incredibly unlikely that the garbage collection would run on a UI thread. — Sheridan, Aug 06 '14 at 11:08
Along with Sheridan's comment, have you profiled your use of memory to see if you can reduce memory consumption? I like the JetBrains suite (no affiliation): [dotMemory](http://www.jetbrains.com/dotmemory/) for memory profiling. — cod3monk3y, Aug 06 '14 at 11:12
You might need .Net 4.5: http://msdn.microsoft.com/library/ee787088%28v=vs.110%29.aspx#background_server_garbage_collection — Matthew Watson, Aug 06 '14 at 11:12
@Sheridan: As stated in my question, analysis of the process dumps (using Debug Diag 2.0) shows that GC is active. I attached the automatically generated warnings to the question. If necessary, you can also have the stack. — floele, Aug 06 '14 at 11:14
@Matthew Watson: Does not install on 2003 R2. So not a possible solution. — floele, Aug 06 '14 at 11:14
@cod3monk3y Memory usage could certainly be reduced, but it's actually meant to keep some large lists of data in memory for performance reasons. I did only a bit of analysis here, but so far it seems like no unnecessary data is kept in memory and it also does not seem to have any "leaks" currently. — floele, Aug 06 '14 at 11:19
The strategy in your second dot point, i.e., to manage garbage collection in your app, is a slippery slope. Instead, consider offloading some of the processing using cloud technology or other distributed technology. — Gayot Fow, Aug 06 '14 at 11:19
@GayotFow It may sound fancy but we have to go without cloud based technology here. — floele, Aug 06 '14 at 11:22
It's been a while since I looked at the function of the GC, but IIRC... large data is fine, it either [goes on the LOH](http://stackoverflow.com/a/4875394/1174169) or gets promoted to Gen 2 if you're keeping it around. That shouldn't be affecting your GC performance. If you've got a long-running, hanging GC, then there's a good possibility that you've got a lot of stuff being collected. Personally, if my GC is hanging, I'm going to profile my use of the GC. — cod3monk3y, Aug 06 '14 at 11:26
@cod3monk3y "Personally, if my GC is hanging, I'm going to profile my use of the GC. " that would make sense if you actually use it manually. But in this case the GC is invoked automatically...so what should I profile there? — floele, Aug 06 '14 at 11:29
Any object you use will go into the GC once you lose all references to it. So you're *always* using the GC. You don't have to make manual calls to it to use it incorrectly. Also, see @Luaan's answer re: CLRProfile and other tools to help understand what your GC is actually doing. — cod3monk3y, Aug 06 '14 at 11:37
Do you have any finalizers or implement the IDispose pattern? Perhaps you're doing a lot of work in a finalizer? [From MSDN](http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic4): "Finally, objects needing finalization create work for the finalizer thread. If your finalization process is a complex one [...] which can cause a backlog of work and therefore cause more objects to linger waiting for finalization. Therefore, it is vitally important that finalizers do as little work as possible." — cod3monk3y, Aug 06 '14 at 11:41
@cod3monk3y I don't think finalizers slow down the collection itself - they're always run in a separate thread. It's just that one finalizer can cause other finalizers to execute way too late (I *think* they're executed sequentially on one thread). So it would be a problem, but not for the collection itself. Although it might be that it would complicate heap compaction... — Luaan, Aug 06 '14 at 11:43
Actually, looking at your stack trace... The GC.Collect is happening when the WinFormsHost is rendering/being layed out (which is scary). It seems from that trace that you actually *are* indirectly forcing a GC via the [circuitous route](http://msdn.microsoft.com/en-us/library/system.gc.addmemorypressure(v=vs.110).aspx) of `GC.AddMemoryPressure`. ([also here](http://stackoverflow.com/a/2618095/1174169)) — cod3monk3y, Aug 06 '14 at 11:51
@cod3monk3y Even worse, the `MemoryPressure.Add` (a WPF method, not `GC.AddMemoryPressure`) is explicitly calling `GC.Collect(2)` - http://referencesource.microsoft.com/#PresentationCore/src/Core/CSharp/MS/Internal/MemoryPressure.cs Ouch. Probably works great with just WPF and the GUI, but once you start doing some serious work in the background, it's deadly. — Luaan, Aug 06 '14 at 11:56
Two notes: I have a WinFormsHost as well, so I'll definitely be checking if this is causing me slow-downs. And, does your WinFormsHost have a transparent background? Does changing it to non-transparent fix the problem? Maybe if you can prevent the chain from `GetBitmapForTransparentWindowsFormsHost` down, and thus a GC during the Layout of your view, the problem will abate. — cod3monk3y, Aug 06 '14 at 11:57
The other thing that piques my interest is `AnimatedRenderMessageHandler`, which ["Occurs for animation processing and updates"](http://msdn.microsoft.com/en-us/library/aa969767(v=vs.110).aspx). So... is this an animated, transparent winformshost in a WPF application that is adding GC memory pressure every step of the animation (by caching a bitmap image of the control)? That could very well cause a huge hiccup in the UI thread. So, try removing any animations on or around the winforms host? — cod3monk3y, Aug 06 '14 at 12:06
@cod3monk3y None of my WindowsFormsHosts have a background color set explicitly (does this qualify as transparent?). I am not animating such controls either, but I may show/hide them dynamically and have some animations at other locations (some fade in/out for menus, nothing special). — floele, Aug 06 '14 at 12:24
In my experience user-triggered Gen-2 GC happens at about 2GB of heap per second. I'm surprised that 500MB of heap would take seconds (corresponding to less than 250MB/sec of heap). — usr, Aug 06 '14 at 12:25
@usr: Could there not be certain more complex GC operations happening in WPF occasionally? As you can see CPU usage is quite intense... — floele, Aug 06 '14 at 12:27
I suggest taking out your WinFormsHosts temporarily and seeing if your problem goes away, based on your stack trace. That's the easiest thing to do, and looks a *lot* like your problem to me. — cod3monk3y, Aug 06 '14 at 12:29
@floele I don't know. I measured it on a 1GB ASP.NET heap with many small objects. Should be even worse than your case. What is your memory usage comprised of? A big bitmap is just a single (or very few) objects for example. Should not generate much GC load. Are you sure GC causes multi-sencond pauses here? Profile the app using PerfView and capture GC info over the course of one day. — usr, Aug 06 '14 at 12:29
WPF controls are by default transparent (Colors.Transparent is different, providing mouse click-ability to transparent areas). If colors and controls *behind* your WinFormsHost (WFH) are showing through, then I'd certainly call that transparent. And I believe WFHs cache their state as bitmaps occasionally [citation needed], which seems to be indicated by your stack trace. — cod3monk3y, Aug 06 '14 at 12:33
@cod3monk3y I checked the framework code a bit and it seems like it doesn't really matter whether the control is painted transparently or opaque. Both cases would lead to `GetRenderTargetBitmapForVisual` which is responsible for `MemoryPressure.Add` and `GC.Collect()` later. — floele, Aug 07 '14 at 12:45
Have you tried taking out the control altogether (just to identify if this *is* or *is not* part of the problem -- it seems a likely candidate since it leads to a GC.Collect in the UI thread)? — cod3monk3y, Aug 07 '14 at 13:41
@cod3monk3y Well, I can't remove those controls for a test in production (and it's hard to reproduce otherwise). I'm on holidays next week, so I won't find time looking into this further at the moment. I'm not quite sure what I do after that, we'll see. — floele, Aug 09 '14 at 08:33

Luaan · Answer 1 · 2014-08-06T15:19:20.017

You can change the GC mode using app.config. By default, server Windows have background GC disabled. The reasoning is quite simple - background GC means less visible latency (optimal for user applications), while foreground GC has better total throughput.

However, if the GC collection is long enough to produce a noticeable hang, you're probably doing something wrong. It isn't really about the amount of memory straight, it's more likely to be a result of inefficient handling of such memory. The by far greatest offender I've seen that does this is using GC.Collect manually - that basically kills all the performance gains of generational GC (among other optimizations), which is extremely important. The Debug Diag snippet seems to indicate this is in fact the case - you seem to be initiating the collection manually; but I've never worked with that tool, so it might be a false positive.

200-500 MiB certainly isn't much. The critical point is how easy is to do the collection. That depends on how the objects get assigned to different generations, how many objects there are (rather than their total size, although that obviously also plays a role), memory locality and many other things.

Somewhat counter-intuitively, in .NET it's usually a bad idea to force yourself to reuse objects and do similar optimizations like in C++. Most likely, it will lead to worse GC performance, because it impairs memory locality and generational partitioning of the heap(s).

The key point still is - profile. Attach Concurrency Visualizer and CLRProfiler. They will tell you how much work the GC is actually doing, and can help you understand why.

And to reiterate - don't use GC.Collect. I've never seen it cause a performance improvement, and I've seen it kill GC performance many, many times. The only reasonable use case I've seen for it was in benchmarks - which are actually simple enough to benefit from forced collection. If you don't attempt those micro-optimizations, the heap can actually be almost as fast as the stack (by mostly working on scope-limited objects - a handy shortcut). .NET's GC is actually very good.

EDIT:

Based on the additional information, I'm affraid this might actually be a significant problem when doing a lot of background work in a WPF application. If you separate the background work into an extra process and only threat the WPF application as a GUI front-end to some underlying service, you might get around the whole problem. Obviously, that's not going to be very easy to implement...

Another option would be to try to limit the WPF garbage as much as possible - the less data WPF works with, the less inclined it will be to call its GC.Collect(2).

The app.config setting to force GC to behave as on non-server windows is quite simple:

<configuration>
 <runtime>
  <gcServer enabled="false" />
  <gcConcurrent enabled="true" />
 </runtime>
</configuration>

It should help driving the time the UI is unresponsive down - provided you have multiple processor cores available.

"you seem to be initiating the collection manually" - no, definitely not. The stack strace shows this is *not* a manual collection (otherwise I wouldn't "complain" about it). — floele, Aug 06 '14 at 11:37
@floele Are you sure? I don't think "natural" collections have `GC.Collect` in their stack. At least I've never seen it there in process dumps / concurrency visualizer / visual studio debugger. — Luaan, Aug 06 '14 at 11:39
As sure as it gets ;) I attached a stack trace now so you can have a look for yourself. — floele, Aug 06 '14 at 11:42
"server Windows have background GC disabled" - Does this apply to all .NET applications running on the server? Do you have a reference where that behaviour is precisely documented? — floele, Aug 06 '14 at 11:43
@floele You don't need to take my word for it - check `GCSettings.IsServerGC` and `GCSettings.LatencyMode`, it will tell you which GC you're actually running. And huh, the stack does make it clear - this is actually happening as part of unmanaged memory allocation, rather than managed memory. It's WPF garbage that's giving you trouble. This might actually be very tricky to solve, since WPF is explicitly calling `GC.Collect(2)`, which is *awful* if you've got other data than just a few forms and images. — Luaan, Aug 06 '14 at 11:49
Thanks, `GCSettings.IsServerGC` is false when running on the server. `GCSettings.LatencyMode` returns "interactive". So that answers a part of my question, I think you should include this information in the answer :) — floele, Aug 06 '14 at 12:45
In regard to the other point you make "I'm affraid this might actually be a significant problem when doing a lot of background work in a WPF application." - who says I'm doing that? The background work in my application is pretty much limited to initially load some lists...everything else pretty much runs on the GUI thread. It's not working continuously on background tasks (other than the GC of course). — floele, Aug 06 '14 at 12:58
@floele You're right, the problem stands whether you're doing work or not - the problem is that the garbage collectors always goes through every single object you've got in memory every time WPF causes a collection. That's more than just annoying - it's quite a deal-breaker. In that case, do try tweaking the GC settings in `app.config` - it might be a significant help. At least it might take the GC latency down to bearable level. This is quite disappointing on WPF's part - yet another example of "even if you're pretty sure `GC.Collect` is a good idea, it probably isn't". — Luaan, Aug 06 '14 at 14:20
What kind of "tweaks" do you have in mind? I never tried making changes to to the GC configuration, not sure which settings are supposed to help me in this case. — floele, Aug 06 '14 at 14:55

Why does background garbage collection sometimes pause my application and how can I prevent it?

1 Answers1