SetWindowsHookEx returns null when hooking to specific thread

Question

I have written following two applications (dll,exe) to hook a dll into putty in order to listen to keyboard events.

One application is Dll application which contains the hook method (meconnect).

#include "stdafx.h"
#include <stdio.h>
#include <windows.h>

INT APIENTRY DllMain(HMODULE hDLL, DWORD Reason, LPVOID Reserved) {
    /* open file */
    FILE *file;
    fopen_s(&file, "C:\\temp.txt", "a+");

    switch (Reason) {
    case DLL_PROCESS_ATTACH:
        fprintf(file, "DLL attach function called.\n");
        break;
    case DLL_PROCESS_DETACH:
        fprintf(file, "DLL detach function called.\n");
        break;
    case DLL_THREAD_ATTACH:
        fprintf(file, "DLL thread attach function called.\n");
        break;
    case DLL_THREAD_DETACH:
        fprintf(file, "DLL thread detach function called.\n");
        break;
    }

    /* close file */
    fclose(file);

    return TRUE;
}

extern "C" __declspec(dllexport) LRESULT  __stdcall meconnect(int code, WPARAM wParam, LPARAM lParam) {
    //FILE *file;
    //fopen_s(&file, "C:\\function.txt", "a+");
    //fprintf(file, "Function keyboard_hook called.\n");
    //fclose(file);

    OutputDebugString(L"function keyboard hook called. \n");
    //return 0;
    return(CallNextHookEx(NULL, code, wParam, lParam));
}

here meconnect method will be called when the keyboard event is fired on PuTTY application.

Given below is the code that will inject above dll into PuTTY application.

// program.exe.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <Windows.h>
#include <string>
#include  <io.h>

using namespace std;

void Usage()
{
    printf("Usage: InjectDLL pid path-to-dll [-privilege]");
}


int _tmain(int argc, char* argv[])
{



    /*
    * Load library in which we'll be hooking our functions.
    */
    HMODULE dll = LoadLibrary(L"C:\\drivers\\dllinject.dll");
    //HMODULE dll = LoadLibrary((LPCTSTR)buf);
    if (dll == NULL) {
        printf("The DLL could not be found.\n");
        getchar();
        return -1;
    }

    /*
    * Get the address of the function inside the DLL.
    */
    HOOKPROC addr = (HOOKPROC)GetProcAddress(dll, "_meconnect@12");
    if (addr == NULL) {
        printf("The function was not found.\n");
        getchar();
        return -1;
    }

    /*
    * Hook the function.
    */
    DWORD procID=0;
    HWND targetWnd = FindWindowA("PuTTYConfigBox","PuTTY Configuration" );
    //HWND windowHandle = FindWindowA(NULL, "Calculator.exe");
    DWORD threadID = GetWindowThreadProcessId(targetWnd, &procID);

    wchar_t msgBuf[1024] = L"";
    wchar_t msgBuf2[1024] = L"";
    wsprintf(msgBuf, L"the proc Id is %d", threadID);

    OutputDebugString(msgBuf);
    HHOOK handle = SetWindowsHookEx(WH_KEYBOARD, addr, dll, threadID);
    DWORD x = GetLastError();
    wsprintf(msgBuf2, L"the last error is %d", x);
    OutputDebugString(msgBuf2);
    if (handle == NULL) {
        printf("The KEYBOARD could not be hooked.\n");
    }
    else{
        printf("Program successfully hooked.\nPress enter to unhook the function and stop the program.\n");
    }

    /*
    * Unhook the function.
    */

    getchar();
    UnhookWindowsHookEx(handle);

    return 0;
}

when I run the above code, setWindowsHookEx always return null, which means SetWindowsHookEx is not hooked into PuTTY application. Could anybody help me to understand what I am doing wrong here.

When `SetWindowsHookEx()` fails, what does `GetLastError()` return? Is the DLL compiled for 32bit or 64bit? Is PuTTY 32bit or 64bit? You can't inject a 32bit DLL into a 64bit process and vice versa. And even if `SetWindowsHookEx()` succeeded, `meconnect()` is declared wrong. The return value needs to be an `LRESULT` instead of an `int` and it must use the `__stdcall` calling convention. — Remy Lebeau, Feb 27 '17 at 07:19
Do improve your error handling, display the value returned by GetLastError(). The `procID` value you pass is not correct, it needs to be a thread id, not a process id. In other words, you need the return value of GetWindowThreadProcessId(). Always best to work from known-good code btw, this isn't quite so easy to get right. — Hans Passant, Feb 27 '17 at 07:20
@RemyLebeau , Thanks for the comments. I have modified the code in the question as per the comments given. After the meconnect is change it says the `The function was not found` . I modified the `meconnect` to have `__stdcall` calling convention and return type is changed to LRESULT. Any thought on why it is not finding the meconnect function — KItis, Feb 27 '17 at 08:01
@HansPassant , Thanks for your comment as well, I have incorporated your comments in the updated code — KItis, Feb 27 '17 at 08:09
you need call `GetProcAddress(dll, "_meconnect@12");` for `x86` and `GetProcAddress(dll, "meconnect");` for x64. if you want undecorated name form for x86 - you need export `meconnect` via def file — RbMm, Feb 27 '17 at 08:09
@RbMm, It worked , thank you all for the help , I have updated the question with working code. — KItis, Feb 27 '17 at 08:21
You don't need a DLL. The [KeyboardProc](https://msdn.microsoft.com/en-us/library/windows/desktop/ms644984.aspx) *"may be called in the context of the thread that installed it."* — IInspectable, Feb 27 '17 at 13:57

RbMm · Accepted Answer · 2017-02-27T08:42:23.470

you need declare function in dll in next way:

extern "C" LRESULT  __stdcall meconnect(int code, WPARAM wParam, LPARAM lParam) {
//...
}

#ifdef _X86_
#define EXP_meconnect "_meconnect@12"
#elif defined(_AMD64_)
#define EXP_meconnect "meconnect"
#else
#error "unknown platform"
#endif

__pragma(comment(linker, "/export:meconnect=" EXP_meconnect))

for always export it by name "meconnect". another way - use def file

or if you declare it as

extern "C" __declspec(dllexport) LRESULT  __stdcall meconnect(int code, WPARAM wParam, LPARAM lParam) {
//...
}

in dll, you need in exe next code:

#ifdef _X86_
#define EXP_meconnect "_meconnect@12"
#elif defined(_AMD64_)
#define EXP_meconnect "meconnect"
#else
#error "unknown platform"
#endif

HOOKPROC addr = (HOOKPROC)GetProcAddress(dll, EXP_meconnect );

SetWindowsHookEx returns null when hooking to specific thread

1 Answers1