Debugging XP SP2/SP3 native application symbol path issue

Question

When I tried to validate, if WinDbg has been setup properly, I opened the executable C:\WINDOWS\NOTEPAD.exe and tried to check the loaded modules.

First of all, these are the environment variables set:

    _NT_SYMBOL_PATH: c:\mysymbols;SRV*c:\symbols*https://msdl.microsoft.com/download/symbols
    _NT_EXECUTABLE_IMAGE_PATH: SRV*c:\symbols

This is the WinDbg output:

CommandLine: C:\WINDOWS\NOTEPAD.EXE
Symbol search path is: SRV*C:\symbols*https://msdl.microsoft.com/download/symbols;c:\mysymbols
Executable search path is: SRV*c:\symbols
ModLoad: 01000000 01014000   notepad.exe
ModLoad: 7c910000 7c9c9000   ntdll.dll
ModLoad: 7c800000 7c908000   C:\WINDOWS\system32\kernel32.dll
ModLoad: 76350000 7639a000   C:\WINDOWS\system32\comdlg32.dll
ModLoad: 77da0000 77e4a000   C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77e50000 77ee3000   C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77fc0000 77fd1000   C:\WINDOWS\system32\Secur32.dll
ModLoad: 773a0000 774a3000   C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202\COMCTL32.dll
ModLoad: 77be0000 77c38000   C:\WINDOWS\system32\msvcrt.dll
ModLoad: 77ef0000 77f39000   C:\WINDOWS\system32\GDI32.dll
ModLoad: 7e360000 7e3f1000   C:\WINDOWS\system32\USER32.dll
ModLoad: 77f40000 77fb6000   C:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 7e670000 7ee91000   C:\WINDOWS\system32\SHELL32.dll
ModLoad: 72f70000 72f96000   C:\WINDOWS\system32\WINSPOOL.DRV
(ef8.f6c): Break instruction exception - code 80000003 (first chance)
eax=001a1eb4 ebx=7ffd5000 ecx=00000007 edx=00000080 esi=001a1f48 edi=001a1eb4
eip=7c91120e esp=0007fb20 ebp=0007fc94 iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll - 
ntdll!DbgBreakPoint:
7c91120e cc              int     3

I switched on sym noisy mode, called .reload and got the continued output when I tried to list the loaded modules:

0:000> !sym noisy
noisy mode - symbol prompts on
0:000> .reload
Reloading current modules
..............
DBGHELP: c:\mysymbols\ntdll.pdb - mismatched pdb
DBGHELP: c:\mysymbols\symbols\dll\ntdll.pdb - file not found
DBGHELP: c:\mysymbols\dll\ntdll.pdb - file not found
SYMSRV:  Die Serververbindung wurde zurückgesetzt.
SYMSRV:  c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb not found
SYMSRV:  https://msdl.microsoft.com/download/symbols/ntdll.pdb/A618C674A4FC40F5B1781029C2C7F68E2/ntdll.pdb not found
DBGHELP: C:\WINDOWS\system32\ntdll.pdb - file not found
DBGHELP: ntdll.pdb - file not found
DBGHELP: Couldn't load mismatched pdb for C:\WINDOWS\system32\ntdll.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll - 
DBGHELP: ntdll - export symbols

0:000> lm
start    end        module name
01000000 01014000   notepad    (deferred)             
72f70000 72f96000   WINSPOOL   (deferred)             
76350000 7639a000   comdlg32   (deferred)             
773a0000 774a3000   COMCTL32   (deferred)             
77be0000 77c38000   msvcrt     (deferred)             
77da0000 77e4a000   ADVAPI32   (deferred)             
77e50000 77ee3000   RPCRT4     (deferred)             
77ef0000 77f39000   GDI32      (deferred)             
77f40000 77fb6000   SHLWAPI    (deferred)             
77fc0000 77fd1000   Secur32    (deferred)             
7c800000 7c908000   kernel32   (deferred)             
7c910000 7c9c9000   ntdll      (export symbols)       C:\WINDOWS\system32\ntdll.dll
7e360000 7e3f1000   USER32     (deferred)             
7e670000 7ee91000   SHELL32    (deferred)

The line 7c910000 7c9c9000 ntdll (export symbols) C:\WINDOWS\system32\ntdll.dll shows, that the symbol files couldn't get loaded successfully.

I have downloaded and installed the symbols (like WindowsXP-KB835935-SP2-slp-Symbols) but still the PDB files don't seem to match with my ntdll.dll build. But why is that the case? Is there any chance I can get this to work?

I see, thank you. In the past, offline symbols have been offered to get downloaded, unfortunately there's no chance to get the old symbols anymore. Have the symbols also been delivered with the old (2005/2010) Visual Studio installation discs? (Tools\Symbols) — kvirk, Feb 28 '22 at 15:47
[xp](http://37.120.179.6/stackoverflow/xp.zip) [2000](http://37.120.179.6/stackoverflow/2000.zip) [2003 xp64](http://37.120.179.6/stackoverflow/2003XP64.zip) — Thomas Weller, Feb 28 '22 at 19:56
Thanks a lot! I've looked into the xp archive, it has xp, sp1 and sp2 symbols. I'm actually using sp3, which one should I use, so that the example I posted above would actually work after I place the symbols into c:\symbols? — kvirk, Feb 28 '22 at 20:14
I've first added the sp2 symbols to the c:\symbols path (with dll subdirectories, etc.), but the pdb was not found. When I placed ntdll.pdb directly in c:\symbols, windbg tried to create a GUID subdirectory with the error, that the pdb inside couldn't be created. Then I placed the pdb in c:\mysymbols and got a mismatched pdb error. While it probably won't work on SP3 with these symbols, what would be the correct way/structure to place the pdbs in, combined with the symbol path? These error messages do feel illogic to me, when the GUID subdirectory is actually created, but file still not found. — kvirk, Feb 28 '22 at 20:57
@ThomasWeller: I even tried the symbol ntdll.pdb (sp2 symbols) on an XP Home SP2 machine, the same message: mismatched pdb. This is frustrating, really frustrating. Do you have a chance to test it yourself? Would it make sense to write to windbgfb@microsoft.com? — kvirk, Mar 01 '22 at 07:08
Thanks for your detailed response and offer to vote for the topic to reopen. It is indeed about XP debugging and I did my post, so that the error even occurs with SP2. — kvirk, Mar 01 '22 at 09:31
If someone has XP SP3 symbols and could provide them for download, I'd gladly put them into my collection. — Thomas Weller, Mar 01 '22 at 14:51

kvirk · Accepted Answer · 2022-03-04T16:05:16.297

As I couldn't stop pondering about the root of the problem, I reread the output above:

SYMSRV:  c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb not found
SYMSRV:  https://msdl.microsoft.com/download/symbols/ntdll.pdb/A618C674A4FC40F5B1781029C2C7F68E2/ntdll.pdb not found

The debugger is looking for a symbol file having the GUID A618C674A4FC40F5B1781029C2C7F68E2, it even outputs an URI it is trying to load the pdb from. Thus, I tried to download the pdb manually, which worked, created the GUID directory manually, renamed the blob file to ntdll.pdb and placed it within the GUID-directory.

As a result, it did finally work, as the following output shows:

:000> .reload
Reloading current modules
............
DBGHELP: c:\mysymbols\ntdll.pdb - mismatched pdb
DBGHELP: c:\mysymbols\symbols\dll\ntdll.pdb - file not found
DBGHELP: c:\mysymbols\dll\ntdll.pdb - file not found
DBGHELP: ntdll - public symbols  
         c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb
..


0:000> lm
start    end        module name
01000000 01014000   notepad    (deferred)             
72f70000 72f96000   WINSPOOL   (deferred)             
76350000 7639a000   comdlg32   (deferred)             
773a0000 774a3000   COMCTL32   (deferred)             
77be0000 77c38000   msvcrt     (deferred)             
77da0000 77e4a000   ADVAPI32   (deferred)             
77e50000 77ee2000   RPCRT4     (deferred)             
77ef0000 77f38000   GDI32      (deferred)             
77f40000 77fb6000   SHLWAPI    (deferred)             
77fc0000 77fd1000   Secur32    (deferred)             
7c800000 7c907000   kernel32   (deferred)             
7c910000 7c9c9000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb
7e360000 7e3f0000   USER32     (deferred)             
7e670000 7ee90000   SHELL32    (deferred)

EDIT: I was finally able to get the automatic download from the symbol server working! I always thought about it in the beginning, but did not believe it could be the reason: With a newer version 6.12.0002.633 it does work as expected, while with the 6.6.07.5 it did not.

Do you have a proxy? Or some firewall which replaces the HTTPS certificate? Have you tried HTTP instead of HTTPS? — Thomas Weller, Mar 03 '22 at 06:38
Nope, the version has been the issue, I updated my answer above. Now it does finally work. — kvirk, Mar 04 '22 at 16:05
Try WinDbg 10 Preview. It's a great improvement over 6.x. 6.6.7.5 is really old — Thomas Weller, Mar 04 '22 at 16:14

Debugging XP SP2/SP3 native application symbol path issue

1 Answers1