16

Most .NET memory profilers that I've tried allow you to take snapshots of memory.

However, I'm trying to diagnose an issue where I'm ending up with huge amounts of memory allocated to .NET that is indicated as "free" by the ANTS profiler. (I've confirmed this problem with other profilers like Mem Profiler and the CLR profiler.

ANTS is showing that I have a large amount of memory fragmentation (100% of free memory with 150MB as the largest chunk.) The total size of all the objects in the heap is 180MB. I have 553 MB allocated to .NET and 152 allocated to "Unmanaged".

However, the size of the Large Object Heap (LOH) is only 175kb. This is the actual size of the objects allocated there. I don't allocate any objects that end up on the LOH permanently.

Hence, my problem, some where along the line, I suspect I am somehow allocating large objects (over the 85k limit for the LOH) and then disposing them.

I'm reading large amounts of data (estimating here at several MB) from databases (Oracle, Sql Server), copying this data to object arrays in memory, and processing the data into indexes (arrays, dictionaries, etc) for easier searching/filtering/processing.

My guess is, the data reader(s) are allocating a lot of space temporarily. However, I don't have a good way to pause the program and take a snapshot of the memory.

What I'd like is a profiler that keeps track of every object allocated on the LOH so I can figure out what is causing the LOH fragmentation and excessive memory usage (the memory is not returned to the OS, so it looks like my process is taking 1GB of memory to store 200MB of allocated objects.) I'm guess the memory is not returned because the LOH is not compacted, so I'm stuck with all this memory for the life of my process, which can be weeks (it runs as a windows service.)

Edit: My problem is that my .NET application is using a lot of memory that I can't trace.

Edit1: I've used the Visual Studio memory profiler. While it does tell me all objects that are instantiated, how many, total bytes, etc, I doesn't give me a hint as to why I end up with so much free memory. My only hint/clue is what ANTS is telling me: "Memory Fragmentation is restricting the size of the objects that can be allocated." and I have a lot of unused memory allocated to .NET that is not used.

Edit2: More profiling shows that I have some short lived large objects allocated on the LOH. However, the total amount allocated on the LOH is never more the 3 to 4 MB. However, during this time the private bytes shoot through the roof, doubling and tripling, while the size of my actually allocated objects (on all heaps) only grows slightly. For instance bytes in all heaps is 115MB but my private bytes are over 512 MB.

ANTS is telling me clearly that I am having a problem with memory fragmentation. Turns out I am creating short lived objects on the LOH. However, these objects never total more than 3 or 4 MB. So these short lived large objects (appear to?) are fragmenting the heck out of the LOH.

To respond to Eric Lippert, and the Disney Land parking lot analogy (which is great).

It's like someone parks in a spot for a few minutes, and then leaves. That spot is then reserved (no one else can park there) until I repave the parking lot!

I first starting investigating this when Visual Studio warned me of memory usage and advised switching to x64. (I forget the warning number, quick google doesn't find it). So switching to x64 alleviates the immediate problem, but doesn't address the underlying problem.

It's like I have a parking lot for 1000 cars, but after I put 100 cars in it, my parking attendants are screaming that it's full...

Luckily I have a huge VMware cluster at my disposal and an understanding admin. I've been allocated 8 cpu's and 8 GB of memory. So as far as a problem, I can deal with it, I just throw resources at it. Also, (as i said above) I switched to x64 a while back as Visual Studio kept nagging me with a warning about However, I'd like to figure out what it allocated on the LOH to see if I an mitigate this Heap fragmentation with some small code changes. Perhaps a fool's errand, given that I can just throw resources at it.

The application runs fine, it's fast with the occasional GC pause. But mostly I can live with the situation, I'd just like to know what objects are causing it. My suspicions are some short lived dictionary's that I haven't tracked down yet.

Edit3: http://msdn.microsoft.com/en-us/magazine/cc188781.aspx

ObjectAllocatedByClass does not track allocations of the large object heap, but ObjectAllocated does. By comparing the notifications from the two, an enterprising soul should be able to figure out what is in the large object heap as opposed to the normal managed heap.

So it looks like this can be done. However, my C++ skills are way to rusty to dig into this (maybe sometime in the future if I get more time). I was hoping that a profiler would provide this out of box.

Chris Weber
  • 5,555
  • 8
  • 44
  • 52
  • Does your application make use of string interning? – Paul Ruane Mar 28 '12 at 16:39
  • 2
    So what specifically is the *problem*? Is it that you're running out of virtual memory, or that the working set is too big and you're thrashing the page file, or what? – Eric Lippert Mar 28 '12 at 16:40
  • When your LOH is only 175kb the fragmentation can't be much of an issue. – H H Mar 28 '12 at 16:55
  • @Paul, no string interning. The only object on the LOH is .NET's interned string.s – Chris Weber Mar 28 '12 at 17:12
  • Whilst I agree with the other comments saying you need to get more specific note that the free Microsoft CLR Profiler can (at great verbosity) log the complete state of the various heaps on every allocation and collection and visualise exactly what is where. The resulting overhead may bring your app to it's knees, but it's free so give it a try. – ShuggyCoUk Mar 28 '12 at 17:14
  • @Henk, yes you would think, but ANTS is telling me that's where a large chunk of memory is going. 100% of free memory is in large fragments, with the largest at 150MB. Obviously this is a lot of free memory. – Chris Weber Mar 28 '12 at 17:15
  • 2
    @EricLippert the problem is that my app is a memory hog and I can't figure out why. – Chris Weber Mar 28 '12 at 17:17
  • @ChrisWeber: Consider the parking lot at DisneyLand. The parking lot being *full* is not a problem in of itself. (In fact if it is *half empty* then Disney probably has a big problem). If the problem is that people are using the DisneyLand parking lot to store their abandoned cars, that's one thing. If the problem is that there are too many improperly parked cars, that's another, and if the problem is that the lot is legitimately too small for the number of people who want to visit the park, that's a third. – Eric Lippert Mar 28 '12 at 17:29
  • 3
    An application using a lot of memory is not a problem in of itself; it is a *concern*, obviously. But if you have enough virtual memory and are not thrashing the disk, then there's no actual *problem* with using a lot of memory that a user would notice. Only solve problems that you actually have. – Eric Lippert Mar 28 '12 at 17:31
  • @ChrisWeber: What are you pulling data out of, and where are you storing this data? Does it go out of scope quickly? Did the link provide any help? – Dominic Zukiewicz Mar 29 '12 at 10:12
  • I'm having the exact same problem. Any luck with a tool or procedure to help you figure out what's going on? Did you resolve your problem? Excellent post by the way. – Mark Apr 29 '13 at 16:06
  • 2
    In my case it turned out that putting the application in GCServer mode caused a lot of "free space" to be allocated to .NET. The reason is that gcserver mode creates heaps per logical processor. I have 8 logical processors and so .NET framework was allocating data onto 8 sets of heaps, which I'm sure causes "over-allocation" and what not. As soon as I turned off gcserver mode, much less "over-allocation" was being done. – Mark May 06 '13 at 19:11
  • @Mark - This was my problem too (or rather, not a problem), thanks. – Narayana Jan 09 '15 at 10:01

2 Answers2

2

After experimenting, I have been able to be notified when the GC is removing things from a generation, but not when its putting it in there.

Since LOH is not generation specific, and there is no specific event I can access for notification of LOH insertions, then the only alternative I can offer is to debug the application, take a crash dump - OR - run it locally and use WinDBG. Here's how you can do it:

  1. Download the Windows SDK 7
  2. Copy the %Microsoft_NET_Framework%\sos.dll to the WinDBG directory
  3. IN WinDBG Click File -> Open Executable -> point to your executable
  4. In the command bar at the bottom type g (Go)
  5. Monitor the memory and when you want to analyse it, go to WinDBG -> Debug menu -> Break
  6. Type load .sos - to load the .NET extensions
  7. Type !dumpheap -min 85000 - this will list large objects, which should be residing on the LOH
         Address               MT     Size
0000000012a17048 000007fee7ae6ae8   400032     
0000000012a78b00 000007fee7ae6ae8   400032     
0000000012ada5b8 000007fee7ae6ae8   400032     
0000000012b3c070 000007fee7ae6ae8   400032     
0000000012b9db28 000007fee7ae6ae8   400032

Next we need to go through each of these and find out whats in them.

  1. Copy the first column (the object address) into the clipboard
  2. Type !do <paste from clipboard>
  3. This will list the contents of the object, its type and size.
CLR Version: 4.0.30319.261
SOS Version: 4.0.30319.239
Name:        System.String
MethodTable: 000007fee7ae6ae8
EEClass:     000007fee766ed68
Size:        400026(0x61a9a) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      8470737076787475867884758166807183888774746571677189..
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee7aec9d0  4000103        8         System.Int32  1 instance           200000 m_stringLength
000007fee7aeb510  4000104        c          System.Char  1 instance               38 m_firstChar
000007fee7ae6ae8  4000105       10        System.String  0   shared           static Empty
                                 >> Domain:Value  000000000055fe50:0000000002a11420 <<

And the lines you want to look for are:

Size:        400026(0x61a9a) bytes
String:      8470737076787475867884758166807183888774746571677189..
  1. Do this for each object

(However, I am assuming it would be a string, so check out the 'Name' property just to be sure. It could be an array.)

Dominic Zukiewicz
  • 8,258
  • 8
  • 43
  • 61
0

Oh god - @EricLippert is in the question - I know I've got this one wrong - but here goes :-)

LOH will only store a specific objects greater than 85k in size. I've found from experience this is usually strings - XML or large enumerable types that were in global variables and/or static variables.

If your LOH is small, you must be storing other objects in memory which most likely end up as Generation 2. These could be global variables like arrays, dictionaries throughout the lifecycle of the application.

So first thing to quickly check is to check if there are any variables that would be present throughout the lifetime of the application? Do these have thousands of entries?

Secondly, .NET has a performance counter statistics for memory size for each generation size. The counters can be pointed directly at an application, so this would be a good next step.

Finally, the GC will only clear out memory when it needs to. So yes, it is possible it would remain the same size if the GC doesn't feel the need (memory pressure wise) to clear out anything. You can force Garbage Collection, but it is generally recommended to let the GC do its job.

I asked a similar question about why a string was being stored in my application for so long. I was using WinDBG to analyse a crash dump I forced by going to Task Manager -> Processes -> Right Click -> Create Crash Dump. Then loaded it into WinDBG (Windows SDK 7).

Hopefully the link will provide you with more guidance to tracking down where the problem is rooted (pun intended).

And a question back to the OP: Is this impacting your server severly or are you just curious as to why the memory is sticking about?

Community
  • 1
  • 1
Dominic Zukiewicz
  • 8,258
  • 8
  • 43
  • 61
  • Dominic, thanks for the info. My problem is, I'm _not_ storing data. The items are only on the LOH temporarily, but the memory allocated is sticking around forever. – Chris Weber Mar 29 '12 at 15:13
  • Gotcha. All I would say really is rest-assured, the GC will clear it out when the memory needs to be removed. sorry to go on about my other post, but I had a very similar problem. WinDBG did list the item on the LOH when I created the crash dump, so I was able to find out what it contained at that moment. If you can get the app running locally, you can attach WinDBG to it and pause the program and examine the LOH yourself. – Dominic Zukiewicz Mar 29 '12 at 15:23
  • 1
    Actually, according to my research the LOH is never compacted. So yes, I can use that memory later if I need it (which I don't) but it just hangs around, and again from my research, will not get released to the OS. https://connect.microsoft.com/VisualStudio/feedback/details/521147/large-object-heap-fragmentation-causes-outofmemoryexception – Chris Weber Mar 29 '12 at 15:35
  • 2
    Its not compacted until you call `System.GC.Collect(GC.MaxGeneration)` or the OS indicates there is high memory pressure, so thats why .NET is just letting it sit about. It is expensive in terms of CPU cycles and memory management but if it makes you sleep at night, give it a go ;-) – Dominic Zukiewicz Mar 30 '12 at 11:11
  • 1
    @DominicZukiewicz, maybe you're confusing the terms. What you named are the cases when the LOH is *collected*. But it is never *compacted* (i.e. the objects in it are moved around) as far as I know in the current implementation. – svick Apr 01 '12 at 00:33