Problems triggering dump on first-chance OutOfMemoryException

Question

I am running across a problem when trying to use DebugDiag to take a dump when a first-chance OutOfMemoryException happens. So I wrote an application that I could use to create an out of memory condition and follow the below instructions:

http://blogs.msdn.com/b/kaushal/archive/2012/05/09/using-debugdiag-to-capture-a-dump-on-first-chance-exception.aspx

But I don't get a first-chance dump, I just get a second-chance dump. When I look at the log files from DebugDiag I get this:

[6/16/2013 9:54:04 PM] First chance exception - 0xe06d7363 caused by thread with  System ID: 4628
[6/16/2013 9:54:04 PM] First chance exception - 0xe0434352 caused by thread with  System ID: 4628
[6/16/2013 9:54:05 PM] Unable to determine CLR exception type

ExceptionObjHexAddr = 0x00000000`00000000

bInnerException = False

DumpObject Output = Invalid parameter 0x00000000`00000000


ChildEBP RetAddr  Args to Child              
002bdee4 6a44c93f e0434352 00000001 00000005 KERNELBASE!RaiseException+0x58
002bdf88 6a573b17 00000000 20b578f4 002be04c clr!RaiseTheExceptionInternalOnly+0x276
002bdfb8 6a5e5589 20b54734 002be090 00000000 clr!UnwindAndContinueRethrowHelperAfterCatch+0x83
002be058 003c0a3a 00000000 00000000 0233d174 clr!JIT_NewArr1+0x1af
...  removed some rows, lots of data ...


OS Thread Id: 0x1214 (0)
Child SP IP       Call Site
002bdfd4 7554c41f [Frame: 002bdfd4] 
002be060 003c0a3a 
...  removed some rows, lots of data ...


Error requesting GC Heap data
Unable to determine bounds of gc heap

Then later on I get this:

[6/16/2013 9:54:05 PM] CLR Exception Type - ''
[6/16/2013 9:54:05 PM] First chance exception - 0xe0434352 caused by thread with  System ID: 4628
[6/16/2013 9:54:05 PM] Unable to determine CLR exception type

Then finally I get

[6/16/2013 9:54:05 PM] CLR Exception Type - ''
[6/16/2013 9:54:05 PM] C:\Windows\Microsoft.NET\Framework\v4.0.30319\diasymreader.dll loaded at 0x615c0000
[6/16/2013 9:54:13 PM] Second chance exception - 0xe0434352 caused by thread with  System ID: 4628

Looks like it may be able to get the exception object's address, it is 0, so when the script calls DumpObject it can't find the exception information.

The way I read these log entries is I get a native first-chance exception from malloc or something, then followed up with the CLR exception for the OutOfMemoryException. I am trying to figure out what the second of the first-chance exceptions are, my code looks like this:

private void OnGrowMemoryCommand(int growMemorySize)
{
    try
    {
        _heldMemoryChunks.Add(new byte[growMemorySize * 1024 * 1024]);
    }
    catch (Exception)
    {
        throw;
    }
    TotalMemorySize += growMemorySize;
}

This code is fired from a command on a WPF button. Because of this, any exceptions that originate from this code should result in a TargetInvocationException, which I believe is the second of the first-chance exception. Then finally from the throw block is the second-chance exception, which is of type TargetInvocationException.

So I start to look at the second-chance dump file. I load it up into windbg, then issue these commands:

.symfix C:\symcache
.loadby sos clr
.reload

!pe
Exception object: 023caf9c
Exception type:   System.Reflection.TargetInvocationException
Message:          Exception has been thrown by the target of an invocation.
InnerException:   System.OutOfMemoryException, Use !PrintException 023c9928 to see more.

I can see that my assumptions above are supported by the fact that the second-chance exception is a TargetInvocationException, but why can't DebugDiag get the CLR exception types? For a sanity check I try a live-debug session. So I start up the application and attach, then I issue these commands.

.symfix C:\symcache
.loadby sos clr
.reload

!threads
Failed to request ThreadStore

!dumpheap
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to 
displaying, finding or traversing objects as well as gc heap segments may not 
work properly. !dumpheap and !verifyheap may incorrectly complain of heap 
consistency errors.
Error requesting GC Heap data
Unable to build snapshot of the garbage collector state

It is totally hosed. So I start to research this issue.

This url suggests it could be multiple CLR instances:

http://blogs.msdn.com/b/jjameson/archive/2011/01/11/issues-debugging-managed-code-in-windbg-with-sos-and-psscor2-e-g-quot-failed-to-request-threadstore-quot.aspx

So I issue these commands:

.cordll
CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll

This is strange to me, I thought that from 4.0 on that mscorwks was ditched in favor of clr. Is mscordacwks 4.5?

I issued this command:

lmvm mscordacwks

But clr is loaded:

lmvm clr
start    end        module name
6a350000 6a9e2000   clr        (pdb symbols)          C:\symcache\clr.pdb\97FD69E1786F42F9A541C81D81AC96852\clr.pdb
    Loaded symbol image file: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
    Image path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
    Image name: clr.dll
    Timestamp:        Fri Mar 29 00:13:44 2013 (51553118)
    CheckSum:         0069496E
    ImageSize:        00692000
    File version:     4.0.30319.18047
    Product version:  4.0.30319.18047
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     clr.dll
    OriginalFilename: clr.dll
    ProductVersion:   4.0.30319.18047
    FileVersion:      4.0.30319.18047 built by: FX45RTMGDR
    PrivateBuild:     DDBLD316
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail

So I don't think I have multiple CLRs loaded.

So I assume that the same thing causing my live debug issues is causing my failure to fire on first-chance problem. Any ideas?

The correct way to get information on this kind of exceptions is to follow this article, http://blogs.msdn.com/b/tom/archive/2008/05/19/asp-net-tips-what-to-gather-to-troubleshoot-part-5-outofmemoryexception.aspx You should not try to get first chance exception dump. Besides, you don't even need to capture a dump at exception, if you can capture hang dumps when memory usage is too high (>1 GB). KB2020006 documents common causes of such problems, so you might start from there, http://support.microsoft.com/kb/2020006 — Lex Li, Jun 17 '13 at 08:53
Looks like you configured DebugDiag to catch first-chance *unmanaged* exceptions. 0xe06d7363 ('msc') is a C++ exception, probably std::bad_alloc. 0xe0434352 ('ccr') is the underlying SEH exception for managed exceptions. This doesn't come alive until you used a managed debugging tool, sos.dll got you the managed exception. — Hans Passant, Jun 17 '13 at 10:10
@Lex Li, there is more than one way to accomplish this and I don't think the one you suggest is appropriate for me. The method requires a registry entry change which I don't want to do on all the boxes. Also, I don't agree that there is not a need to capture a dump on the OOM exception, I agree you can see the problems prior to that but it is nice to know what is going on when the OOM occurs. Lastly, I still can't debug a live instance with windbg which is of greater concern than the dump. — taylorjonl, Jun 17 '13 at 14:59
@Hans Passant, I only have the CLR exception configured with a filter for the OOM exception, it dumps a row for each first-chance exception to the log because it has to determine the type of the exception to see if it should do the dump. — taylorjonl, Jun 17 '13 at 15:36

score 2 · Accepted Answer · answered Sep 23 '13 at 20:33

I came across the same error on two machines. Both were equipped with .NET 4.5, so I think this is the reason it did not work.

It turned out to be a bug in DebugDiag Scripts. It fails to retrieve the name of the CLR exception in one of it's scripts. However, this problem is fixable, since every DebugDiag rule creates a vbs script, which can be modified. It uses the very same commands, you can also use in WinDbg. Here is how to fix it:

Create your DebugDiag rule using the GUI, as you already did.
For every rule, there is a vbs script file generated. Open it in a text editor under C:\Program Files\DebugDiag\Scripts\CrashRule_<rulename>.vbs
Find the function GetCLRExceptionType

Modify it as follows:

Function GetCLRExceptionType(ByVal ExceptionObjHexAddr, ByVal bInnerException)
    Dim Output, Lines, i

    If Debugger.IsClrExtensionMissing Then
        WriteToLog "Unable to determine CLR exception type - extension dll could not be loaded."
    Else
        ' Output = Debugger.Execute("!DumpObj " & ExceptionObjHexAddr) ' Does not work in .NET 4.5
        Output = Debugger.Execute("!pe") ' FIX .NET45

        Lines = Split(Output, Chr(10))
        For i = 0 To UBound(Lines)      
            If bInnerException Then
                If InStr(Lines(i), "_innerException") <> 0 Then
                    Tokens = Split(Lines(i), " ")
                    For j = 0 To UBound(Tokens)
                        If Len(Tokens(j)) = 8 Then                              
                            GetCLRExceptionType = GetCLRExceptionType(Tokens(j), False)
                            Exit For
                        End If
                    Next
                End If
            ElseIf Len(Lines(i)) >= 7 Then
                If InStr(Lines(i), "Exception type:") = 1 Then  ' FIX .NET45
                    GetCLRExceptionType = Trim(Mid(Lines(i), 16))  ' FIX .NET45
                    Exit For
                End If
            End If
        Next

        If GetCLRExceptionType = "" Then
            If g_ClrExceptionTypeFailureLogCount < MAX_CLR_EXCEPTION_TYPE_FAILURE_LOG_ENTRIES Then
                g_ClrExceptionTypeFailureLogCount = g_ClrExceptionTypeFailureLogCount + 1

                WriteToLog "Unable to determine CLR exception type" & vbcrlf & _
                    "ExceptionObjHexAddr = " & ExceptionObjHexAddr & vbcrlf & _
                    "bInnerException = " & bInnerException & vbcrlf & _
                    "DumpObject Output = " & Output & vbcrlf & _
                    Debugger.Execute("kb100") & vbcrlf & _
                    Debugger.Execute("!clrstack") & vbcrlf & _
                    Debugger.Execute("!dso")
            End If
        End If
    End If
End Function

By default, !DumpObj is used to dump the content of the exception object. However, something changed in .NET 4.5 and obviously, the type of the exception cannot be extracted anymore. Instead, place the !pe command here and parse the exception type from it's result.

In order to make !pe work, you need to do .loadby sos clr. I added it to the Sub Debugger_OnLoadModule:

Sub Debugger_OnLoadModule(ByVal NewModule)
    WriteToLog NewModule.ImageName & " loaded at " & Debugger.GetAs32BitHexString(NewModule.Base)
    Select Case UCase(NewModule.ModuleName)
        Case "MSCORWKS", "MSCORSVR", "CLR", "CORECLR"
            UpdateDeferredManagedBreakpoints
    End Select

    Debugger.Execute(".loadby sos clr")
End Sub

After making the changes you need to either restart the processs or deactivate/reactivate the rule to apply it.

Also: be aware, that if you modify your rule in the DebugDiag GUI, you changes will be overwritten completely.

Great answer! Works for me. On the same note - I just upgraded to DebugDiag 2.0, in which I didn't encounter this issue — M.A. Hanin, Jan 07 '14 at 15:58

Problems triggering dump on first-chance OutOfMemoryException

1 Answers1