Debug python dump file in windbg

Question

one of my python.exe (v3.5.2) process hang in remote Windows Server. I cannot attach it from remote site so I create a full memory dump and download it back to analyze.

however, windbg doesn't have the tool like py-bt in gdb. what I can do is downloading cpython symbol files and loading into windbg.

the question is: how can I print the python script call stack just like "py-bt" do?

the call stack is as follow, looks like hang on HTTPS connection

0:000> k
ChildEBP RetAddr  
0050dd24 74592242 ntdll!ZwWaitForSingleObject+0xc
0050dd6c 7459bdee mswsock!SockWaitForSingleObject+0x143
0050ddf8 777561e0 mswsock!WSPRecv+0x391
0050de48 614d8e76 ws2_32!recv+0x103
0050de68 614138bb _ssl!sock_read+0x26 [c:\build\cpython\externals\openssl-1.0.2h\crypto\bio\bss_sock.c @ 141]
0050de90 614bc229 _ssl!BIO_read+0x5b [c:\build\cpython\externals\openssl-1.0.2h\crypto\bio\bio_lib.c @ 210]
0050debc 614bc36e _ssl!ssl3_read_n+0x1c9 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 255]
0050df78 614bd136 _ssl!ssl3_get_record+0x9e [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 339]
0050dfb8 614c0bb5 _ssl!ssl3_read_bytes+0x1a6 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 1228]
0050dfdc 614c0c13 _ssl!ssl3_read_internal+0x45 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_lib.c @ 4459]
0050dfec 61402880 _ssl!ssl3_read+0x13 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_lib.c @ 4483]
0050e034 613ff8f7 _ssl!_ssl__SSLSocket_read_impl+0x1f0 [c:\build\cpython\modules\_ssl.c @ 1944]
0050e080 61b12ae3 _ssl!_ssl__SSLSocket_read+0x67 [c:\build\cpython\modules\clinic\_ssl.c.h @ 267]
0050e09c 61b7409f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109]
0050e0d0 61b712a5 python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4705]
0050e148 61b7301f python35!PyEval_EvalFrameEx+0x20a5 [c:\build\cpython\python\ceval.c @ 3239]
0050e194 61b74259 python35!_PyEval_EvalCodeWithName+0x75f [c:\build\cpython\python\ceval.c @ 4018]
0050e1dc 61b740f3 python35!fast_function+0x109 [c:\build\cpython\python\ceval.c @ 4813]
0050e210 61b712a5 python35!call_function+0x353 [c:\build\cpython\python\ceval.c @ 4730]
0050e288 61b7301f python35!PyEval_EvalFrameEx+0x20a5 [c:\build\cpython\python\ceval.c @ 3239]

Your best bet is going be using [`gdbinit`](https://github.com/python/cpython/blob/v3.5.2/Misc/gdbinit) as an example for how to get frame data. Start by looking at the stack frame `python35!PyEval_EvalFrameEx+0x20a5`. There should be a variable named [`co`](https://github.com/python/cpython/blob/v3.5.2/Python/ceval.c#L810) on the stack. From there, you can see `co_name`, `co_filename` which are compact `PyASCIIObject`s representing the module and filename. If you can figure out how v3.5.2 represents it, `co_lnotab` should let you figure out the line number. — Sean Cline, Mar 30 '17 at 16:33
For me, `?? (char*)((PyASCIIObject*)(co->co_name)+1)` and `?? (char*)((PyASCIIObject*)(co->co_filename)+1)` print the module and file, since [compact `PyASCIIObject`s store their payload in the same allocation](https://github.com/python/cpython/blob/v3.5.2/Include/unicodeobject.h#L480), but after the struct itself. Hopefully that's enough to at least give you an idea where in the python code things are hung. — Sean Cline, Mar 30 '17 at 16:38

score 15 · Accepted Answer · answered Jun 03 '17 at 20:27

I was surprised that nothing like CPython's gdbinit existed for WinDbg so I decided to write a debugger extension. I've uploaded the code to the PyExt repository on GitHub.

To install it:

Grab the latest zip from the releases page
Unzip pyext.dll into WinDbg's winext directory. (Make sure to use the x86 or x64 version that corresponds to the build of WinDbg you're using.)
Open a dump file in WinDbg and load the extension with: .load pyext
Add the Python symbol server to the symbol path with: .sympath+ srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/
Use !pystack to see the Python backtrace at the time the dump was captured.

Example:

0:000> .load pyext
0:000> .sympath+ srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/

************* Symbol Path validation summary **************
Response  Time (ms)     Location
Deferred                srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Deferred                srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/

0:000> ~*e!pystack
Thread 0:
    File "C:\python33\lib\threading.py", line 284, in wait
        [Globals] 
    File "C:\python33\lib\threading.py", line 1028, in join
        [Globals] 
    File "scripts\win32debug.py", line 153, in _launch_and_wait
        [Globals] 
    File "scripts\win32debug.py", line 180, in dump_process
        [Globals] 
    File "object_types.py", line 33, in <module>
        [Locals] [Globals]

If you run into any problems or would find a new feature to be helpful, feel free to open an GitHub issue and let me know.

Thanks :-] it saves lots of my time to find out python script stack — SILENCE, Jun 20 '17 at 02:05
Note: if `.load pyext` fails with `The call to LoadLibrary(pyext) failed, Win32 error 0n126 "The specified module could not be found."` despite `pyext.dll` being in the correct place, then you probably need to install Microsoft Visual C++ Redistributable for Visual Studio 2017 (or possibly 2015?) from https://support.microsoft.com/en-gb/help/2977003/ because pyext.dll depends on `MSVCP140.dll` and `VCRUNTIME140.dll` — sparrowt, Jan 23 '19 at 14:58
It works with IDAPython too (attach to IDA Pro with WinDbg). Thanks! — Pavel Sapehin, Oct 03 '19 at 08:38

Debug python dump file in windbg

1 Answers1