SetWindowsHookEx is injecting 32-bit DLL into 64-bit process and vice versa

Question

I've been working on an application that requires monitoring thread specific mouse activity (WH_MOUSE) on another process and encountered something very curious.

After finding out that this is not possible via exclusively managed code if I don't want to use WH_MOUSE_LL and that I'd need a native DLL export to inject itself in the target process, I set out and created it in C++ according to what scattered documentation I could find on the subject and then tried using it to hook into Notepad.

Although according to GetLastWin32Error the injection succeeded, I was not getting notified of mouse events. After nearly giving up and going for the low level global hook option, I re-read the "Remarks" section of this article which made me suspect that the problem may be because of the "bitness" of my code vs notepad:

A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL cannot be injected into a 32-bit process. If an application requires the use of hooks in other processes, it is required that a 32-bit application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit processes, and a 64-bit application call SetWindowsHookEx to inject a 64-bit DLL into 64-bit processes.

However, both my native DLL and managed application were compiled as x64, and I was trying to hook into the 64-bit version of notepad, so it should've worked fine, but I took a shot in the dark anyway and went into the SysWOW64 folder and opened the 32-bit Notepad from there, tried hooking in again and this time the hook worked beautifully!

Curiously, I then recompiled both my native DLL and managed app as x86 and tested it it against the 32-bit Notepad and it didn't work, but it worked on my normal 64-bit Notepad!

How am I possibly seem to be able to inject a 32-bit DLL into a 64-bit process and vice versa!

Although my original problem has been solved and I can continue with my app's development, the curiosity as to why I'm observing this strange inverse behavior from SetWindowsHookEx is driving me insane, so I really hope someone will be able to shed some light on this.

I know this a lot of talk and no code, but the code for even a sample app is rather large and comes in both managed and unmanaged flavors, however I'll promptly post any piece of the code you think might be relevant.

I've also created a sample app so you can test this behavior yourself. It's a simple WinForms app that tries to hook into Notepad and displays its mouse events:

http://saebamini.com/HookTest.zip

It contains both an x86 version and an x64 version. On my machine (I'm on a 64-bit Windows 7), the x86 version only works with 64-bit Notepad, and the x64 version only works with 32-bit Notepad (from SysWOW64).

UPDATE - Relevant Bits of Code:

C# call to the unmanaged library:

public SetCallback(HookTypes type)
{
    _type = type;

    _processHandler = new HookProcessedHandler(InternalHookCallback);
    SetCallBackResults result = SetUserHookCallback(_processHandler, _type);
    if (result != SetCallBackResults.Success)
    {
        this.Dispose();
        GenerateCallBackException(type, result);
    }
}

public void InstallHook()
{
    Process[] bsProcesses = Process.GetProcessesByName("notepad");
    if(bsProcesses.Length == 0)
    {
        throw new ArgumentException("No open Notepad instance found.");
    }
    ProcessThread tmp = GetUIThread(bsProcesses[0]);

    if (!InitializeHook(_type, tmp.Id))
    {
        throw new ManagedHooksException("Hook initialization failed.");
    }
    _isHooked = true;
}

[System.Runtime.InteropServices.DllImport("user32.dll")]
private static extern int GetWindowThreadProcessId(IntPtr hWnd, IntPtr procid);

// 64-bit version
[DllImport("SystemHookCore64.dll", EntryPoint = "InitializeHook", SetLastError = true,
        CharSet = CharSet.Unicode, ExactSpelling = true,
        CallingConvention = CallingConvention.Cdecl)]
private static extern bool InitializeHook(HookTypes hookType, int threadID);


[DllImport("SystemHookCore64.dll", EntryPoint = "SetUserHookCallback", SetLastError = true,
        CharSet = CharSet.Unicode, ExactSpelling = true,
        CallingConvention = CallingConvention.Cdecl)]
private static extern SetCallBackResults SetUserHookCallback(HookProcessedHandler hookCallback, HookTypes hookType);

C++:

HookProc UserMouseHookCallback = NULL;
HHOOK hookMouse = NULL;

HINSTANCE g_appInstance = NULL;

MessageFilter mouseFilter;

bool InitializeHook(UINT hookID, int threadID)
{
    if (g_appInstance == NULL)
    {
        return false;
    }

    if (hookID == WH_MOUSE)
    {
        if (UserMouseHookCallback == NULL)
        {
            return false;
        }

        hookMouse = SetWindowsHookEx(hookID, (HOOKPROC)InternalMouseHookCallback, g_appInstance, threadID);
        return hookMouse != NULL;
    }
}

int SetUserHookCallback(HookProc userProc, UINT hookID)
{   
    if (userProc == NULL)
    {
        return HookCoreErrors::SetCallBack::ARGUMENT_ERROR;
    }

    if (hookID == WH_MOUSE)
    {
        if (UserMouseHookCallback != NULL)
        {
            return HookCoreErrors::SetCallBack::ALREADY_SET;
        }

        UserMouseHookCallback = userProc;
        mouseFilter.Clear();
        return HookCoreErrors::SetCallBack::SUCCESS;
    }

    return HookCoreErrors::SetCallBack::NOT_IMPLEMENTED;
}

int FilterMessage(UINT hookID, int message)
{
    if (hookID == WH_MOUSE)
    {
        if(mouseFilter.AddMessage(message))
        {
            return HookCoreErrors::FilterMessage::SUCCESS;
        }
        else
        {
            return HookCoreErrors::FilterMessage::FAILED;
        }
    }

    return HookCoreErrors::FilterMessage::NOT_IMPLEMENTED;
}

static LRESULT CALLBACK InternalMouseHookCallback(int code, WPARAM wparam, LPARAM lparam)
{
    if (code < 0)
    {
        return CallNextHookEx(hookMouse, code, wparam, lparam);
    }

    if (UserMouseHookCallback != NULL && !mouseFilter.IsFiltered((int)wparam))
    {
        UserMouseHookCallback(code, wparam, lparam);
    }

    return CallNextHookEx(hookMouse, code, wparam, lparam);
}

Answer 1

My best guess about your problem :

The Windows hook system is able to hook both 32-bit and 64-bit application, from any bitness. The thing is, as you pointed, you can't inject a DLL into an application with the wrong bitness. To make this work, Windows will normally inject the DLL if it can, but if it can't, it will setup a callback that use the hooking application message loop. Since the message loop is handled by the OS, it is used to make a call from different bitness.

In your case, the only thing that work is the message loop way. And there is a good reason for that : your 64-to-64 and 32-to-32 calls have no chance to succeed, because the hook is in the injected DLL, that is, in a different process than your application.

Nothing happens in your case because your UserMouseHookCallback stay to NULL. Indeed, the call to SetUserHookCallback() is done in the application DLL instance, but UserMouseHookCallback is unchanged in the target DLL instance. Once injected, the DLL is in a different process, and should be considered as such. You have to find another way to call back the application (maybe post a message, like in the 32-to-64 case, and/or make use of shared sections).

To test this, put something like MessageBox() in InternalMouseHookCallback(). The box should appear even in 64-to-64 and 32-to-32.