Stacktrace of an inner exception

Question

w3wp process hosting my .NET application is crashing at random times. I have collected a dump file by setting up a second chance exception rule using DebugDiag. Here are the steps I have performed.

The lastevent command shows a .NET exception.

0:027> .lastevent
Last event: 1ae4.2e98: CLR exception - code e0434352 (first/second chance not available)

The stack trace of this thread looks as follows,

0:027> !CLRStack
OS Thread Id: 0x2e98 (68)
        Child SP               IP Call Site
000000266c1bab18 00007fff6e5d95fc [HelperMethodFrame: 000000266c1bab18] 
000000266c1bac00 00007fff5cb38afb System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bcb48 00007fff6657120d [HelperMethodFrame: 000000266c1bcb48] 
000000266c1bcc30 00007fff5cb359b5 System.Runtime.AsyncResult.Complete(Boolean)
000000266c1bea00 00007fff6657120d [FaultingExceptionFrame: 000000266c1bea00] 
000000266c1bef00 00007fff5bca63a5 System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)
000000266c1bef60 00007fff5cb3586c System.Runtime.AsyncResult.Complete(Boolean)
000000266c1befd0 00007fff570da192 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)
000000266c1bf020 00007fff5cb38b63 System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bf090 00007fff5cb38ac7 System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bf0f0 00007fff650a045c System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [f:\dd\ndp\clr\src\BCL\system\threading\overlapped.cs @ 135]
000000266c1bf2a0 00007fff66416793 [GCFrame: 000000266c1bf2a0] 
000000266c1bf498 00007fff66416793 [DebuggerU2MCatchHandlerFrame: 000000266c1bf498] 
000000266c1bf628 00007fff66416793 [ContextTransitionFrame: 000000266c1bf628] 
000000266c1bf858 00007fff66416793 [DebuggerU2MCatchHandlerFrame: 000000266c1bf858]

pe command shows a callback exception

0:027> !pe
Exception object: 0000002552364d38
Exception type:   System.Runtime.CallbackException
Message:          Async Callback threw an exception.
InnerException:   System.NullReferenceException, Use !PrintException 0000002552364ae8 to see more.
StackTrace (generated):
    SP               IP               Function
    000000266C1BCC30 00007FFF5CB359B5 System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0x235
    000000266C1BEFD0 00007FFF570DA192 System_ServiceModel_Activation_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x92
    000000266C1BF020 00007FFF5CB38B63 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x53
    000000266C1BF090 00007FFF5CB38AFB System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x6b
    000000266C1BF0F0 00007FFF650A045C mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x7c

StackTraceString: <none>
HResult: 80131501
There are nested exceptions on this thread. Run with -nested for details

Per output of previous command, I look for nested exceptions

0:027> !PrintException -nested /d 0000002552364ae8
Exception object: 0000002552364ae8
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    000000266C1BEF00 00007FFF5BCA63A5 System_Web_ni!System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)+0x55
    000000266C1BEF60 00007FFF5CB3586C System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0xec

StackTraceString: <none>
HResult: 80004003

Nested exception -------------------------------------------------------------
Exception object: 0000002552364d38
Exception type:   System.Runtime.CallbackException
Message:          Async Callback threw an exception.
InnerException:   System.NullReferenceException, Use !PrintException 0000002552364ae8 to see more.
StackTrace (generated):
    SP               IP               Function
    000000266C1BCC30 00007FFF5CB359B5 System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0x235
    000000266C1BEFD0 00007FFF570DA192 System_ServiceModel_Activation_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x92
    000000266C1BF020 00007FFF5CB38B63 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x53
    000000266C1BF090 00007FFF5CB38AFB System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x6b
    000000266C1BF0F0 00007FFF650A045C mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x7c

StackTraceString: <none>
HResult: 80131501

My next step is to find the stack trace for that NullReferenceException

0:027> !do 0000002552364ae8
Name:        System.NullReferenceException
MethodTable: 00007fff652865a0
EEClass:     00007fff64c3f180
Size:        160(0xa0) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff65276948  400028e        8        System.String  0 instance 000000244f54d350 _className
00007fff6528f6b8  400028f       10 ...ection.MethodBase  0 instance 0000002552366f58 _exceptionMethod
00007fff65276948  4000290       18        System.String  0 instance 0000000000000000 _exceptionMethodString
00007fff65276948  4000291       20        System.String  0 instance 000000244bf27668 _message
00007fff65286788  4000292       28 ...tions.IDictionary  0 instance 00000025523751a8 _data
00007fff65276b78  4000293       30     System.Exception  0 instance 0000000000000000 _innerException
00007fff65276948  4000294       38        System.String  0 instance 0000000000000000 _helpURL
00007fff65276f28  4000295       40        System.Object  0 instance 0000002552364cc0 _stackTrace
00007fff65276f28  4000296       48        System.Object  0 instance 0000000000000000 _watsonBuckets
00007fff65276948  4000297       50        System.String  0 instance 0000000000000000 _stackTraceString
00007fff65276948  4000298       58        System.String  0 instance 0000000000000000 _remoteStackTraceString
00007fff65279288  4000299       88         System.Int32  1 instance                0 _remoteStackIndex
00007fff65276f28  400029a       60        System.Object  0 instance 0000000000000000 _dynamicMethods
00007fff65279288  400029b       8c         System.Int32  1 instance      -2147467261 _HResult
00007fff65276948  400029c       68        System.String  0 instance 000000255242ce60 _source
00007fff6528fbc0  400029d       78        System.IntPtr  1 instance                0 _xptrs
00007fff65279288  400029e       90         System.Int32  1 instance       -532462766 _xcode
00007fff65250340  400029f       80       System.UIntPtr  1 instance     7fff5bca63a4 _ipForWatsonBuckets
00007fff65265538  40002a0       70 ...ializationManager  0 instance 0000002552364c40 _safeSerializationManager
00007fff65276f28  400028d       b8        System.Object  0   shared           static s_EDILock
                                 >> Domain:Value  000000244acba390:NotInit  000000267072bd80:NotInit  <<

Here I attempt to get the stack trace of NullReferenceException. This looks like a SByte array.

0:027> !do 0000002552364cc0
Name:        System.SByte[]
MethodTable: 00007fff65202b20
EEClass:     00007fff64c34f60
Size:        120(0x78) bytes
Array:       Rank 1, Number of elements 96, Type SByte (Print Array)
Content:     ........0Qkv&....c.[.......l&...x..[............kX.\....`..l&...`..\............................
Fields:
None

My expectation is to get a stacktrace/details of what method/line of code/object is responsible for causing this null reference. I also attempt to dump the contents of SByte array but that doesn't provide me any useful information. Any suggestions on how can I get more information about this NullReferenceException?

Why are you dumping the NRE as an object? You should try !sos.pe 0000002552364ae8 first. Also, note that !sos.pe will give you the last exception on the current thread. You may want to check !sos.threads to see if any other treads have relevant (recent) exceptions. Also, you can dump the managed heap for all exceptions. Something like !sos.dumpheap -type Exception . Though note that older exceptions will have been GCed so may no longer exist. Also, there are a few pre-allocated exceptions (OutOfMemory, StackOverlow, ExecutionEngine, ThreadAbort) that probably didn't occur, so be careful inter. — Dono, Feb 09 '18 at 10:25

score 1 · Accepted Answer · answered Feb 09 '18 at 14:29

First of all, I think that !pe -nested already shows the call stack that you're looking for:

0:027> !PrintException -nested /d 0000002552364ae8
[...]
StackTrace (generated):
    SP               IP               Function
    000000266C1BEF00 00007FFF5BCA63A5 System_Web_ni!System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)+0x55
    000000266C1BEF60 00007FFF5CB3586C System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0xec
[...]

Next, a stack trace is exactly what you see in that SByte[]: it's just a bunch of numbers.

Those number do get their meaning only together with PDB files. The PDB file contains the information to turn numbers into names.

Have a look at the values in the SP column and IP column. They are:

000000266C1BEF00
000000266C1BEF60 
00007FFF5BCA63A5 
00007FFF5CB3586C

And now, use a converter that converts those values into text:

You'll find that the printable characters resssemble those of the output shown in SByte[] by WinDbg:

0:027> !do 0000002552364cc0
[...]
Content:     ........0Qkv&....c.[.......l&...x..[............kX.\....`..l&...`..\............................

The order may be different due to little/big-endianness.

Stacktrace of an inner exception

1 Answers1