2

There is a keyboard hook installed like this:

s_hKeyboardHook = ::SetWindowsHookEx(WH_KEYBOARD, KeyboardHookProc, nullptr, ::GetCurrentThreadId());

(This is a plug-in that wants to intercept keyboard events that get sent to its host (64-bit), even though the host doesn't provide keyboard events to its plugins the normal way. I do not have the source code of the host, though I do have the source code of the plug-in.)

After the keyboard hook procedure successfully runs and returns, the program crashes. The crash happens inside Windows' ZwCallbackReturn(), executing the syscall instruction. The exception is 0XC0000005 (access violation). The crash only happens if a particular key is pressed which triggers some particular logic.

This is how it looks like in the debugger (animated gif)

I am stuck diagnosing this crash and could really use some help. I am sure the problem is in this big chunk of code that's in the hook proc. What I am having trouble with is understanding where the crash occurs and where to basically place the breakpoint to preempt it.

Additional info:

1) The hook procedure is really really heavy, with lots of blocking, i/o and memory usage (it completes in a couple of seconds on a fast machine). Maybe that's part of the problem.

2) If compiled as 32-bit, the stack right after the crash looks more interesting, but I doubt it can be trusted:

2a71f510()  Unknown
ExecuteHandler2@20()    Unknown
ExecuteHandler@20() Unknown
_RtlDispatchException@8()   Unknown
_KiUserExceptionDispatcher@8()  Unknown
2a10f24a()  Unknown
_DispatchHookW@16() Unknown
_CallHookWithSEH@16()   Unknown
___fnHkINDWORD@4()  Unknown
_KiUserCallbackDispatcher@12()  Unknown
_LdrAddLoadAsDataTable@20() Unknown
AfxInternalPumpMessage() Line 153   C++
AfxWinMain(0x00000000, 0x00000020, 0x00000001, 1638280) Line 47 C++
@BaseThreadInitThunk@12()   Unknown

where the top 5 lines are repeated many times.

Here's what I tried so far. It is my understanding that the syscall instruction itself doesn't generate the exception: the registers look sane, and I guess the stack would remain the same if it crashed. So I think that after this instruction initiates transition back to the kernel mode, from where the "user callback" (the hook procedure call) had originated, the kernel continues to run just fine. Eventually it should return control back to userland -GetMessage() I presume). Then down the road, I think, the stack gets corrupted and the program crashes. But unfortunately I can't instruct my Visual C++ debugger to break at the first user-mode instruction executed, before the stack is corrupted. I tried installing conditional breakpoints in TranslateMessage() and DispatchMessage(), which are most likely to run right after GetMessage(), but they don't fire between the last good user-mode instruction and the crash.

user38329
  • 719
  • 4
  • 17
  • `The crash only happens if a particular key is pressed which triggers some particular logic` - this is a bit vague for us to be able to discern what the problem might be. – 500 - Internal Server Error May 16 '16 at 20:41
  • If you strip all the logic out of your hook and simply return immediately, does it still crash? If not, you know where to look. – Jonathan Potter May 16 '16 at 20:58
  • Is the plugin running in the same process as the host? If yes, I assume the plugin is a DLL that gets loaded explicitly. Is it possible the plugin DLL is being unloaded, perhaps in response to the keypress? – Adrian McCarthy May 16 '16 at 21:55
  • Yes I am sure the problem is in this big chunk of code that's in the hook proc. What I am having trouble with is understanding where the crash occurs and where to basically place the breakpoint to preempt it. – user38329 May 17 '16 at 09:54

1 Answers1

2

The crash happened because the keyboard hook procedure was NOT the first in the hook chain. It was called from a previous hook in the hook chain via CallNextHookEx(). And that previous hook was registered by a DLL which got unloaded inside "our" keyboard hook.

Therefore, after all the hooks got eventually called, the control returned to the first hook procedure, which didn't exist any more. And the crash was trying to execute an invalid address.

user38329
  • 719
  • 4
  • 17