Global CBT hook is never called

Question

I'm trying to make an application that gets informed on creation and destruction of top-level windows, system-wide. I've made a code to exploit a CBT hook. The solution contains two projects, DLL and EXE. EXE project has a reference to the DLL project. The hook is being set from a DLL. There is a message loop in EXE project. The problem is that CBT hook is not working. With the help of VS debugger, I've found out that hook callback is never called, while return code from SetWindowsHookEx is non-zero, implying the hook was set. What's a mistake? How do I fix it?

Here's a minimal example. DLL, main.cpp:

#include <windows.h>

typedef void(*DECODERPROC)(int code, WPARAM wParam, LPARAM lParam);

HINSTANCE hInst = nullptr;
HHOOK hHook = nullptr;
DECODERPROC fpDecoder = nullptr;

LRESULT CALLBACK cbtProc(int code, WPARAM wParam, LPARAM lParam) {
    // FIXME: never called
    if (code > 0 && fpDecoder) {
        fpDecoder(code, wParam, lParam);
    }
    return CallNextHookEx(hHook, code, wParam, lParam);
}

__declspec(dllexport) bool InstallHook(DECODERPROC decoder) {
    if (hHook) return false;
    fpDecoder = decoder;
    return (hHook = SetWindowsHookEx(WH_CBT, cbtProc, hInst, 0)) != NULL;
}

__declspec(dllexport) bool UninstallHook() {
    if (!hHook) return false;
    bool res = UnhookWindowsHookEx(hHook) != NULL;
    if (res) hHook = NULL;
    return res;
}

BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) {
    hInst = reinterpret_cast<HINSTANCE>(hModule);
    return TRUE;
}

EXE, main.cpp

#include <iostream>
#include <locale>

#include <windows.h>
#include <fcntl.h>
#include <io.h>
using namespace std;

typedef void(*DECODERPROC)(int code, WPARAM wParam, LPARAM lParam);
__declspec(dllimport) bool InstallHook(DECODERPROC);
__declspec(dllimport) bool UninstallHook();

int main() {
    _setmode(_fileno(stdout), _O_U8TEXT);
    WNDCLASS windowClass = {};
    windowClass.lpfnWndProc = [](HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) -> LRESULT {
        if (message == WM_DESTROY)
            UninstallHook();
        return DefWindowProc(hWnd, message, wParam, lParam);
    };
    LPCWSTR windowClassName = L"Foobar";
    windowClass.lpszClassName = windowClassName;
    if (!RegisterClass(&windowClass)) {
        wcerr << L"Failed to register window class" << endl;
        return 1;
    }
    HWND messageWindow = CreateWindow(windowClassName, 0, 0, 0, 0, 0, 0, HWND_MESSAGE, 0, 0, 0);
    if (!messageWindow) {
        wcerr << L"Failed to create message-only window" << endl;
        return 1;
    }

    InstallHook([](int code, WPARAM wParam, LPARAM lParam) {
        wcout << L"Never called" << endl;
    });

    MSG msg;
    while (GetMessage(&msg, 0, 0, 0) > 0) {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }
    return msg.wParam;
}

Answer 1

If you are running on Windows 64-bit, you need both 32-bit and 64-bit versions of the DLL in order to hook every running process. A 32-bit DLL cannot hook a 64-bit process, and vice versa. So you need to call SetWindowsHookEx() from a 32-bit process to hook 32-bit processes, and from a 64-bit process to hook 64-bit processes.

More importantly than that, a separate instance of your DLL gets injected into every running process, so your fpDecoder callback pointer is NULL in every instance of the DLL except for the one that your EXE is calling InstallHook() on. So you need to redesign your hook to use inter-process communication (window message, named pipe, mailslot, socket, etc) to communicate with your main EXE, you can't use a function pointer.

Depending on which process you are actually debugging, you might not see cbtProc() getting called. If you are debugging your main EXE process, your code is not doing anything to trigger any CBT activity within your EXE's process once the hook has been installed, and the debugger would not show you any CBT activity occurring in other processes that it is not debugging.

Depending on what you are actually looking for in your hook, you might consider using SetWinEventHook() instead, as it can be used with or without a DLL, and it has more flexible filtering capabilities than SetWindowsHookEx().