1

We have setup procdump as the (AeDebug) postmortem debugger to capture dumps of unhandled exceptions. The registry key is set to "c:\my\sysinternals\procdump.exe" -accepteula -ma -j "c:\dumps" %ld %ld %p

Currently I'm looking at a dump where the process that triggered a crash dump is still running hours after dumping the process has completed?!
It was my assumption that any process that triggers a crash dump is about to get terminated?

From WinDbg 

Debug session time: Tue Dec  1 01:53:06.000 2015 (UTC + 1:00)
System Uptime: 18 days 18:09:24.556
Process Uptime: 1 days 0:09:31.000

0:000> ~
.  0  Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
...

0:000> ? 178c
Evaluate expression: 6028 = 0000178c

From ProcExp

Looking at PID 6028 on Tue Dec 1 10:00:00, hours after the dump from 01:53:06.000

<3thParty>.exe:6028 Properties
Started: 1:43:35   30/11/2015

Edit

0:000> ~*kb1000

.  0  Id: 178c.eb4 Suspend: 0 Teb: fffdd000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0018de38 77cd8567 00000574 00000001 00000000 ntdll!ZwWaitForSingleObject+0x15
01 0018debc 77cd8695 0018e0c8 0018e118 00000001 ntdll!RtlReportExceptionEx+0x14b
02 0018df14 772f8dd8 0018e0c8 0018e118 00000001 ntdll!RtlReportException+0x86
03 0018df34 772f8edb 0018dfa4 0018dfa4 00000000 ole32!SilentlyReportExceptions+0x79
04 0018df48 772f94e5 0018dfa4 00000000 00000000 ole32!ServerExceptionFilter+0xbb
05 0018df60 77360827 0018dfa4 0148502c 01048240 ole32!AppInvokeExceptionFilterWithMethodAddress+0x11
06 0018df7c 77183f21 00000000 0018faf8 77278760 ole32!SyncStubInvoke+0x82
07 0018df90 77183eae 00000000 00000000 00000000 msvcrt!_EH4_CallFilterFunc+0x12
08 0018dfbc 77286771 7736662c 7726ea10 0018e0c8 msvcrt!_except_handler4_common+0x8e
09 0018dfdc 77c8b499 0018e0c8 0018fae8 0018e118 ole32!_except_handler4+0x20
0a 0018e000 77c8b46b 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler2+0x26
0b 0018e024 77c8b40e 0018e0c8 0018fae8 0018e118 ntdll!ExecuteHandler+0x24
0c 0018e0b0 77c40133 0118e0c8 0018e118 0018e0c8 ntdll!RtlDispatchException+0x127
0d 0018e0b0 00404880 0118e0c8 0018e118 0018e0c8 ntdll!KiUserExceptionDispatcher+0xf
WARNING: Stack unwind information not available. Following frames may be wrong.
0e 0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
0f 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
10 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
11 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
12 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
13 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
14 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
15 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
16 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
17 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
18 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
19 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
1a 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
1b 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
1c 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
1d 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
1e 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
1f 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
20 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
21 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
22 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
23 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
24 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
25 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
26 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
27 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b

   1  Id: 178c.17a4 Suspend: 0 Teb: fffda000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 023cfed4 76b614ab 00000180 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
01 023cff40 77661194 00000180 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 023cff58 77661148 00000180 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 023cff6c 76bb511d 00000180 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
04 023cff88 7766338a 00234b88 023cffd4 77c69f72 ws2_32!SockAsyncThread+0x25
05 023cff94 77c69f72 00234b88 2e876a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 023cffd4 77c69f45 76bb50f8 00234b88 ffffffff ntdll!__RtlUserThreadStart+0x70
07 023cffec 00000000 76bb50f8 00234b88 00000000 ntdll!_RtlUserThreadStart+0x1b

   2  Id: 178c.1108 Suspend: 0 Teb: fffd7000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0273fdf4 77c82f91 00000003 010169d0 00000001 ntdll!NtWaitForMultipleObjects+0x15
01 0273ff88 7766338a 00000000 0273ffd4 77c69f72 ntdll!TppWaiterpThread+0x33d
02 0273ff94 77c69f72 010169a0 2ec86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0273ffd4 77c69f45 77c82e65 010169a0 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0273ffec 00000000 77c82e65 010169a0 00000000 ntdll!_RtlUserThreadStart+0x1b

   3  Id: 178c.1794 Suspend: 0 Teb: fffac000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0908fe28 77c83392 000001c0 0908fedc 25b36ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0908ff88 7766338a 01015748 0908ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0908ff94 77c69f72 01015748 25b36a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0908ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0908ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b

   4  Id: 178c.5e4 Suspend: 0 Teb: fffaf000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 08b0fe28 77c83392 000001c4 08b0fedc 240b6ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 08b0ff88 7766338a 01015748 08b0ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 08b0ff94 77c69f72 01015748 240b6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 08b0ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 08b0ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b

   5  Id: 178c.c80 Suspend: 0 Teb: fffa9000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0b03fed8 76b63bd5 00000000 0b03ff1c 57962de4 ntdll!ZwDelayExecution+0x15
01 0b03ff40 76b644a5 0000ea60 00000000 0b03ff78 KERNELBASE!SleepEx+0x65
02 0b03ff50 7724d98d 0000ea60 010bd460 7724cd48 KERNELBASE!Sleep+0xf
03 0b03ff5c 7724cd48 00000000 00000000 010bd460 ole32!CROIDTable::WorkerThreadLoop+0x14
04 0b03ff78 7724d87a 00000000 00000000 0b03ff94 ole32!CRpcThread::WorkerLoop+0x26
05 0b03ff88 7766338a 010bd460 0b03ffd4 77c69f72 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x16
06 0b03ff94 77c69f72 010bd460 27b86a95 00000000 kernel32!BaseThreadInitThunk+0xe
07 0b03ffd4 77c69f45 7724d864 010bd460 ffffffff ntdll!__RtlUserThreadStart+0x70
08 0b03ffec 00000000 7724d864 010bd460 00000000 ntdll!_RtlUserThreadStart+0x1b

   6  Id: 178c.14f4 Suspend: 0 Teb: fffa6000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0b13fe28 77c83392 000001bc 0b13fedc 27a86ac9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
01 0b13ff88 7766338a 01015748 0b13ffd4 77c69f72 ntdll!TppWorkerThread+0x216
02 0b13ff94 77c69f72 01015748 27a86a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0b13ffd4 77c69f45 77c83e85 01015748 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0b13ffec 00000000 77c83e85 01015748 00000000 ntdll!_RtlUserThreadStart+0x1b

   7  Id: 178c.ee4 Suspend: 0 Teb: fffa3000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0b23fcbc 76b614ab 00000578 00000000 0b23fd04 ntdll!ZwWaitForSingleObject+0x15
01 0b23fd28 77661194 00000578 00002710 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
02 0b23fd40 77661148 00000578 00002710 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
03 0b23fd54 6e22b3d6 00000578 00002710 00000000 kernel32!WaitForSingleObject+0x12
04 0b23ff88 7766338a 01072188 0b23ffd4 77c69f72 comsvcs!PingThread+0x131
05 0b23ff94 77c69f72 01072188 27986a95 00000000 kernel32!BaseThreadInitThunk+0xe
06 0b23ffd4 77c69f45 6e22b320 01072188 ffffffff ntdll!__RtlUserThreadStart+0x70
07 0b23ffec 00000000 6e22b320 01072188 00000000 ntdll!_RtlUserThreadStart+0x1b

   8  Id: 178c.914 Suspend: 0 Teb: fff9d000 Unfrozen
 # ChildEBP RetAddr  Args to Child
00 0d26ff5c 72be635c 00000534 0d26ff90 0d26ff84 ntdll!ZwRemoveIoCompletion+0x15
01 0d26ff88 7766338a 72be64b3 0d26ffd4 77c69f72 mswsock!SockAsyncThread+0x83
02 0d26ff94 77c69f72 0101d3e8 219d6a95 00000000 kernel32!BaseThreadInitThunk+0xe
03 0d26ffd4 77c69f45 72be62ee 0101d3e8 ffffffff ntdll!__RtlUserThreadStart+0x70
04 0d26ffec 00000000 72be62ee 0101d3e8 00000000 ntdll!_RtlUserThreadStart+0x1b

Edit 2

0:000> .exr -1
ExceptionAddress: 00404880 (<3thParty>+0x00004880)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000000
Attempt to read from address 00000000

0:000> ~* e s -d esp L0xffff 0x1003f
0018e118  0001003f 00000000 00000000 00000000  ?...............
0018e9a8  0001003f 00000000 00000000 00000000  ?...............
0018efa8  0001003f 00000000 00000000 00000000  ?...............

0:000> .cxr 0018e118  
eax=00fc2c80 ebx=00000000 ecx=00000000 edx=00fc2c50 esi=00000000 edi=00000000
eip=00404880 esp=0018e400 ebp=0018f61c iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
<3thParty>+0x4880:
00404880 8b11            mov     edx,dword ptr [ecx]  ds:002b:00000000=????????
0:000> kbnf
  *** Stack trace for last set context - .thread/.cxr resets it
 #   Memory  ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00           0018f61c 75a15932 01485018 01549420 00000000 <3thParty>+0x4880
01        24 0018f640 75a905f1 00523859 0018f828 00000004 rpcrt4!Invoke+0x2a
02       404 0018fa44 7735aec1 010ac688 0101fb58 0104ea78 rpcrt4!NdrStubCall2+0x2ea
03        48 0018fa8c 76acffd3 010ac688 0104ea78 0101fb58 ole32!CStdStubBuffer_Invoke+0x3c
04        24 0018fab0 7735d876 010ac688 0104ea78 0101fb58 oleaut32!CUnivStubWrapper::Invoke+0xcb
05        48 0018faf8 7735ddd0 0104ea78 01048240 01095448 ole32!SyncStubInvoke+0x3c
06        4c 0018fb44 77278a43 0104ea78 010be578 01097470 ole32!StubInvoke+0xb9
07        dc 0018fc20 77278938 0101fb58 00000000 01097470 ole32!CCtxComChnl::ContextInvoke+0xfa
08        1c 0018fc3c 7727950a 0104ea78 00000001 01097470 ole32!MTAInvoke+0x1a
09        2c 0018fc68 7735dccd 0104ea78 00000001 01097470 ole32!STAInvoke+0x46
0a        34 0018fc9c 7735db41 d0908070 0101fb58 01097470 ole32!AppInvoke+0xab
0b        e0 0018fd7c 7735e1fd 0104ea20 0101fef0 00000400 ole32!ComInvokeWithLockAndIPID+0x372
0c        28 0018fda4 77279367 0104ea20 00000400 00ff3548 ole32!ComInvoke+0xc5
0d        14 0018fdb8 77279326 0104ea20 00000000 77279286 ole32!ThreadDispatch+0x23
0e        44 0018fdfc 76eb62fa 00e2019e 00000400 0000babe ole32!ThreadWndProc+0x161
0f        2c 0018fe28 76eb6d3a 77279286 00e2019e 00000400 user32!InternalCallWinProc+0x23
10        78 0018fea0 76eb77c4 00000000 77279286 00e2019e user32!UserCallWinProcCheckWow+0x109
11        60 0018ff00 76eb788a 77279286 00000000 00e2019e user32!DispatchMessageWorker+0x3bc
12        10 0018ff10 004aea6a 0018ff34 00010001 0018ff70 user32!DispatchMessageW+0xf
13        1c 0018ff2c 004aeaaf 00e2019e 00000400 0000babe <3thParty>+0xaea6a
14        24 0018ff50 00a0c705 0018ff78 00a0c733 0018ff70 <3thParty>+0xaeaaf
15        20 0018ff70 00a3024b 0018ffc4 00404cac 0018ff88 <3thParty>!FrobWidget+0x4db279
16        18 0018ff88 7766338a fffde000 0018ffd4 77c69f72 <3thParty>!FrobWidget+0x4fedbf
17         c 0018ff94 77c69f72 fffde000 2ca36a95 00000000 kernel32!BaseThreadInitThunk+0xe
18        40 0018ffd4 77c69f45 00a30220 fffde000 ffffffff ntdll!__RtlUserThreadStart+0x70
19        18 0018ffec 00000000 00a30220 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
Lieven Keersmaekers
  • 57,207
  • 13
  • 112
  • 146
  • Very unclear what puzzles you. If you hope that procdump kills the process *before* it takes the dump then, no, that cannot work. – Hans Passant Dec 01 '15 at 12:00
  • @HansPassant - It was my assumption that the process that got dumped would have been killed after the dump and any postprocessing that might occur completed. The comment Thomas gave could be an explanation about why it's still running *"and then the unhandled exception handler prevents the app from dying"* – Lieven Keersmaekers Dec 01 '15 at 12:07
  • Use `~*kb1000` to check to see if any of the threads are blocked waiting for Windows Error Reporting or blocked waiting for I/O to complete. – Marc Sherman Dec 01 '15 at 14:21
  • @MarcSherman - I have added the output of `~*kb1000` but I'm not sure why you suspect a blocking issue?! ftr - `!busy` shows thread 5 being, well, not waiting. – Lieven Keersmaekers Dec 01 '15 at 14:38
  • The process took an unhandled exception at `<3thParty>+0x4880`. It is still running at the time of the dump because once a process is terminated, it loses all its memory, and that would render the dump useless. Once the dump is complete, the process is terminated if the unhandled exception is fatal. If the process continues to run after the dump is complete, then my guess is that the exception was nonfatal. Look at the exception record to see what the exception was. – Raymond Chen Dec 01 '15 at 14:45
  • Single best description of Windows exception handling to be found anywhere IMO https://www.microsoft.com/msj/0197/exception/exception.aspx – JJF Dec 01 '15 at 14:49
  • @RaymondChen - I have added the exception to the question. `.exr -1` shows an Access violation. I also tried searching for exception records and changed the context to the first one and rerun the stack output. *(hope it will help offcourse)* – Lieven Keersmaekers Dec 01 '15 at 14:54
  • @JJF - Read that in the past but a very good refresher, tx. – Lieven Keersmaekers Dec 01 '15 at 15:02
  • 1
    @RaymondChen - I should really rephrase my question I think. Both you and Hans Passant seem to be thinking that my problem is that the process is still running *at* the time of the dump. My problem is that the process is still running hours *after* the dump has been taken. Your comment about fatal and nonfatal exceptions might be what I'm after. I have to read up on that. – Lieven Keersmaekers Dec 01 '15 at 15:09
  • Using a modified version of [this demo program](https://stackoverflow.com/questions/34031107/how-does-exception-dispatching-change-with-an-unhandled-exception-handler) I can now say that AeDebug does not come into play if there is an unhandled exception handler. Just add a `while(true){}` look into the handler. – Thomas Weller Dec 01 '15 at 22:16

1 Answers1

3

Normally, an unhandled exception does cause the process to be terminated once the dump has been captured. What's happening here is that you are running afoul of a compatibility behavior: If a COM server crashes while handling an inbound request, COM historically "handles" the crash by abandoning the call, returning RPC_E_SERVERFAULT to the external caller, and then pretending that the crash never occurred.

That's why the process is still running after the crash dump is taken: The exception was handled. Mind you, it was "handled" by ignoring it, but technically it was handled.

I would recommend using IGlobalOptions to set the COMGLB_EXECPTION_HANDLING property to one of the COMGLB_EXCEPTION_DONOT_HANDLE... values.

Raymond Chen
  • 44,448
  • 11
  • 96
  • 135
  • Which makes the order of exception handling even more interesting... Do I get it right? First the OS considered the exception as unhandled; it goes through AEDebug and LocalDump settings, starts the debugger, which took the dump; The debugger returned control to the OS again, which would normally terminate the process, but for some reason the OS now thinks COM should handle it? This does not sound right – Thomas Weller Dec 01 '15 at 21:14
  • 3
    The exception first goes to COM, since it is the closest exception handler. COM says "Oh, I'm going to handle this exception, but first I will make a special call to the OS to say 'Hey, if anybody wants to take a dump of the process, now is a good time.' That's what SilentlyReportExceptions does. Once the dump is complete, COM says "Okay, all handled" and execution continues. – Raymond Chen Dec 02 '15 at 01:22
  • @RaymondChen - tx. I wouldn't have frobbed this out for the life of me – Lieven Keersmaekers Dec 02 '15 at 06:25
  • Thanks very much. I'd like to reproduce that "special call to the OS". Is it a documented WinAPI call? Which one? – Thomas Weller Dec 03 '15 at 05:23
  • @ThomasWeller It looks like you can use [ReportFault](https://msdn.microsoft.com/en-us/library/windows/desktop/bb513615(v=vs.85).aspx), though I haven't tried it myself. – Raymond Chen Dec 03 '15 at 18:21