Debugging/troubleshooting windows applications without source code

Question

A few days back I faced this problem at work and I was wondering is there a way to mint out more data from the scenario than to go to Microsoft. There has been many cases like this and I wanted to explore that as a windows developer what is the best / optimum way to fetch most of info is such case. I ll describe the situation:

1. An office application, while printing with certain settings (involving cmyk color space) throws up a dialog box error which has a bad description."File %s could not be opened because it is locked by some other application". It does not give a file name nor does the event viewer. Printing is aborted.

2. on using procmon, we find "file locked error" on several files when the api CreatefileMapping is called by involved processes like the office app, the spooler, the splwow64.exe (yes its a 64 bit system and the app is 32 bit).

3. the problem is not there when there is no splwow64 is involved, means using 64 bit app on 64 bit os.

I want to know that which tools will be useful for getting more info in such situations. This includes use of MS symbols with windbg and debugging assembly if needed. Basically i need the file name that is locked , which is shown as %s and the root of the problem.

Answer 1

Debugging without source code and without symbols is hard. The approach to be taken varies much with what you can observe and what assumptions you can make. The following sounds reasonable for your situation.

Diff the 64 bit file accesses against 32 bit version

Since the 64 bit version works, see what the difference in file accesses is. You can export Process Monitor logs to CSV and write a tool which analyzes the file accesses. Your analysis tool should probably

• summarize multiple file read accesses with different offsets to a single file access for the whole file.
• ignore the read order, because the order seems less important.
• ignore DLLs at first

Maybe you can even do that in Excel itself.

Capturing file access calls

First of all I think using Process Monitor is already a good approach. You find out what the application is doing without knowing anything about it, so that would be my first choice.

Of course it's hard to narrow down the list of files in question. If necessary you need to go through all of them, e.g. with the help of a batch file and find a command line tool which can help finding the application which has opened the file. In this case, look at SysInternals Handle. It is a command line utility and you can give it a file name. Basically, it's the command line version of Process Explorer (see below).

Reading files and writing files is usually done through API calls, so API Monitor is another option. Filter for all file access methods which are suspect (ReadFile, ReadFileEx, LockFile, LockFileEx, WriteFile, WriteFileEx, ...).

This can also be achieved in WinDbg by setting breakpoints, but you might hit them very often, so handling them probably needs some automation. You can specify a command string for the bp command.

Be lucky

It might happen that the file name which was intended to be used as argument for the %s format string is somewhere in memory.

1. Take a crash dump at the time the application shows the message
2. Use the SysInternals Strings utility to dump all strings of the dump.
3. Either filter that output directly for C:\, D:\ etc by piping it to the find command or redirect it to a text file for later analysis

This will still leave you with a lot of files, so finding the culprit is not much easier than with the previous approach. Again, narrow the list down with other tools, e.g. handle.

1. Take a crash dump at the time the application shows the message
2. Open the dump in WinDbg
3. Dump strings for some of the addresses you find on the stack. Since file names in NTFS are Unicode, du <address> should work fine here.

This definitely needs more knowledge on internal stuff, e.g. where to find good pointers etc.

Analyzing First Chance Exceptions

If the application relies on exceptions, you might get some insight by checking for first chance exceptions. However, if the application does a good precondition check, there will be less exceptions, thus lowering your chances to grab something useful.

If the situation reproduces well, you can

1. Set up WinDbg as the debugger for that process using Image File Execution Options.
2. Open a logfile with .logopen /t /u c:\firstchanceexceptions.log
3. Let all exceptions be dumped using sxe -c ".exr -1;k;g" *. The g in that command will immediately continue so that the application does (hopefully) not run into timeouts.
4. Continue execution with g
5. When the application terminates, close the logfile with .logclose
6. Review the log file for interesting exceptions / interesting call stacks
7. Reproduce the error but stop on interesting exceptions this time
8. Look at method parameters and memory whether you find the file name (again, use du <address>)

Finding the application which belongs to a message box

If you can't figure out which application shows the error message, e.g. because of a bad message title, do this:

1. Download SysInternals Process Explorer and run it.

2. Drag the cross hair from the toolbar over the window.

3. Process Explorer now highlights the executable which owns the window

Finding the locking application if the file name is printed

If the dialog shows the file name and not just "%s":

1. Also use Process Explorer

2. Use Find/Find handle or DLL or press Ctrl+F

3. Type the file name or parts of the file name