2

Crash Dump Debugging

Suspension of Disbelief: I know your first instinct may be to announce that "It isn't possible!" since this isn't how MS suggests solving the problem and they don't make tools available to accomplish it (easily). Forget that hurdle for a moment and ask yourself, "Is there some way to do this?"

I have a Windows binary I support in production that was built with debug information (MSVC 14.1 /Zi) but the debugger reports "was not built with debug information." This not wholly true—I think it was stripped by some tool during a post-build/pre-release process that I do not control, because I do have a PDB file for it. I have a memory dump from a run (crash) of that executable. Visual Studio and WinDbg both claim that the module has "no debug info," though WinDbg will attempt to load symbols for it anyway, if it finds some that match.

DBGHELP: No debug info for MyDll.dll.  Searching for dbg file
DBGHELP: r:\crashes\symbols\MyDll.dbg - file not found
DBGHELP: r:\crashes\symbols\dll\MyDll.dbg - path not found
DBGHELP: r:\crashes\symbols\symbols\dll\MyDll.dbg - path not found
DBGHELP: MyDll.dll missing debug info.  Searching for pdb anyway
*** WARNING: Unable to verify checksum for MyDll.dll
DBGENG:  MyDll.dll has mismatched symbols - type ".hh dbgerr003" for details
DBGHELP: MyDll - private symbols & lines 
        r:\crashes\symbols\MyDll.pdb - unmatched

Here's the result of !chksym MyDll in WinDbg

MyDll.dll
    Timestamp: 5CF5ABF8
  SizeOfImage: 2437000
      pdb sig: 0
          age: 0

Loaded pdb is r:\crashes\symbols\MyDll.pdb

MyDll.pdb
      pdb sig: 944F882B-73AE-45D0-9043-44C899BE09C5
          age: 1

sig MISMATCH: MyDll.pdb and MyDll.dll

Is there a way to modify the memory dump (or the binary—assuming I can reproduce the problem) so as to re-inject whatever is necessary for the debugger to acknowledge that the symbols file I have matches the binary?

If your answer is "Nope!" I would appreciate some references to support that conclusion.

In the end, I want to be able to make sense of the call stack. Looking at variables/memory would be cool, too, but the call stack itself would be a gold mine in terms of debugging this problem.

Is "debug info" in this regard just a PE header (i.e. revealed by dumpbin /HEADERS) or is it more? Can I inject this header back into the binary/module? Is there more information that the debugger needs besides the symbols file to make sense of function addresses?

The tools for resolving symbol mismatches seem to work by modifying the PDB file. In my case, that doesn't seem to work because the binary module has no PDB signature in it to match to.

References

Corollary Questions

This is a problem I would like to solve one way or another (debugging a crash dump from a run of this "stripped" executable). So, it's possible that I'm asking the wrong question. Is there another way to solve this problem?

  • Can I decode memory addresses "outside" the Visual Studio debugger using the data in the PDB file?
  • Is there some other way to have the debugger use the symbols in a PDB file as if there was a match (assuming the module "has no debug info")?
mojo
  • 4,050
  • 17
  • 24
  • try providing .symopt+ 40 (SYMOPT_LOAD_ANYTHING) flag to let windbg load the partial matched pdb – blabb Jul 13 '19 at 22:07

1 Answers1

2

If you can patch the PDB signature GUID and the age field from the PDB back into the binary, that might be enough to convince the tools that you have the right symbols. It seems likely those fields (or the section they live in) were cleared by the stripping tool.

The LLVM project has some documentation on the PDB format. Specifically, this section has some brief mentions about how PDBs and binaries are matched by the debuggers. And this section may tell you how to find (or recreate) the PDB Guid and age fields in the binary.

There are also some flags in the PE header (like IMAGE_FILE_LINE_NUMS_STRIPPED) that signal the debugger that information has been stripped. You might have to reset those to convince the debugger to even try to look for the PDB.

I don't know what else the stripping tool may have removed. In 32-bit binaries, I believe the frame-pointer opimization (FPO) data that's required for reliable stack unwinding is included in both the executable and the PDB. If the FPO data was stripped from the executable, I'd expect the debugger to use the data from the PDB, but I've not confirmed that. If I had to guess, I'd expect Windbg is more likely to rely on the PDB than the VS debugger.

In x86_64 binaries, basic stack unwinding is simpler (e.g., you should almost always be able to get a stack trace without much unwinding data). But if you want to examine locals and arguments, you do need some unwinding data, and that's stored in the PDB.

Sorry I have only clues rather than answers. I hope this is helpful.

Update 1: From your ChkMatch link, I read:

ChkMatch is capable of making an executable and PDB file match if they have different signatures but the same age (see this article for more information about PDB signature and age). If the age differs, the tool cannot make the files match. Signature and age can be displayed using -c option.

In the display you quoted, the ages are different (0 in executable and 1 in PDB). This seems the likely cause of why chkmatch -m won't solve your problem.

Adrian McCarthy
  • 45,555
  • 16
  • 123
  • 175
  • I would also expect `chkmatch` to fail to help because it modifies the PDB and not the executable: "When ChkMatch is invoked with -m option, it attempts to modify the debug information file." – mojo Jul 12 '19 at 20:08
  • Also, are all executables "signed" in a way that would thwart a simple binary edit (say changing four bytes) would cause Windows to balk at running the exe? – mojo Jul 12 '19 at 20:11
  • If an executable is signed with an Autheniticode signature (to ensure the user that it's from a trusted source), then I imagine changing the executable would break it. If you're talking about the PDB signature, no, that's just a GUID (generated at link time) used to help determine whether you have the right PDB for the given executable. I don't recall, but the minidump might also contain the PDB GUID. Hmm. – Adrian McCarthy Jul 12 '19 at 20:26
  • I'm only asking about the executable. If I change something, it'll have to be the executable/memory-dump. I think that's the place where I have to change something to "help" the debugger draw the correct conclusion. – mojo Jul 12 '19 at 21:00