First things first, you've gotta track down what is actually happening. In situations like this, my first tool is always WinDbg.
http://www.windbg.org/
http://en.wikipedia.org/wiki/WinDbg
To use it with managed/.NET code, you'll need to use SOS (Son of Strike) extension:
http://msdn.microsoft.com/en-us/library/bb190764.aspx
http://blogs.msdn.com/b/johan/archive/2007/11/13/getting-started-with-windbg-part-i.aspx
So, once you've attached WinDbg to your w3wp.exe process, the first thing you'll want to do is figure out what is actually in your heaps:
!dumpheap -stat
This will give you a nicely formatted view of all the currently "live" objects in memory, along with how many of them there are, how many bytes they are taking up, grouped by object type.
Now, the Large Object Heap (LOH) - so normally, as objects are garbage collected, a compaction occurs, kind of like defragmenting a hard drive. This keeps allocations for new objects fast and efficient. Problem is, large objects are not easy to compact - everything has to move around to accomodate them. So, anything that takes up more than 85000 bytes is stuck in a special place called the Large Object Heap. This heap is NOT compacted, so over time, much like a hard drive, fragmentation occurs, leaving unused gaps in the heap, which leads to the runtime needing more space, etc, etc.
So, let's ask windbg to tell us what is in the LOH:
!dumpheap -stat -min 85000
This will show you what is actually in the Large Object Heap - some of these objects may jump right out at you, like a List or a MyClass[].
IMPORTANT: If the things you see in the Large Object Heap are intentionally long-lived (like a static instance of a logger, for example), it's probably not really a problem. You do want to try and keep down the number of short-lived/often created objects in there, however, to reduce fragmentation.
So, I recommend a cheat sheet for SOS exploration:
http://windbg.info/doc/1-common-cmds.html
http://windbg.info/doc/2-windbg-a-z.html
Fun commands:
!gcroot <address> <- will show you what object is "rooting" another
!CLRStack <- show current managed call stack
!dumpobj <address> <- show information about object at address
But my all-time favorite is:
bp clr!SVR::gc_heap::allocate_large_object "!CLRStack; g;"
Which sets a breakpoint on the actual internal call the CLR uses when allocating an object on the large object heap that, when hit, will dump out a full stack trace, then continue.