Hooking with MS Detours crash when hook function is called C++

Question

I wrote a simple program that does 3 things:

It calls MessageBoxA (MBA) with the text "NOT HOOKED"

Then it loads a dll file that I've created that hooks the MBA function and recalls MBA with the text "HOOKED".

After that, it calls MBA again with the same text ("NOT HOOKED"). Of course, the second MBA call should be hooked and display a message with "HOOKED" text it.

Eventually it calls FreeLibrary and exits.

Here is the .cpp file:

#include <iostream>
#include <Windows.h>

using namespace std;

int main()
{
    //Place the path of the dll file here, "DLLs\\HookDLL.dll" is the default path.
    char dllPath[] = "HookDLL.dll"; 

    //Display a pop-up message with the "NOT HOOKED" message and title. 
    MessageBoxA(NULL, "NOT HOOKED", "NOT HOOKED", MB_OK);

    //Load the dll file
    HMODULE hModule = LoadLibraryA((LPCSTR)dllPath);

    //If hModule is null, then the dll wasn't loaded.
    //An error message will be printed out to the console.
    if (!hModule) {
        cout << "Couldn't load the DLL file!" << endl;

        return 1;
    }

    //This is the tricky part.
    //This should display a pop-up message like before with the "NOT HOOKED" message and title,
    //but the dll that was loaded should hook MessageBoxA function,
    //and call a new one with a "HOOKED" message and title instead.
    MessageBoxA(NULL, "NOT HOOKED", "NOT HOOKED", MB_OK);

    FreeLibrary(hModule);

    return 0;
}

here is the .dll file:

#include "pch.h"
#include "detours.h"
#include <iostream>
#include <Windows.h>

using namespace std;

typedef int(WINAPI* MBA)(HWND, LPCSTR, LPCSTR, UINT);

MBA originalMBA = NULL;

int HookedMessageBoxA(
    HWND   hWnd,
    LPCSTR lpText,
    LPCSTR lpCaption,
    UINT   uType
) {
    return originalMBA(NULL, "HOOKED", "HOOKED", MB_OK);
}

BOOL APIENTRY DllMain(HMODULE hModule,
    DWORD  reason,
    LPVOID lpReserved
)
{
    if (reason == DLL_PROCESS_ATTACH) {
        DetourRestoreAfterWith();
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());

        originalMBA = (MBA)DetourFindFunction("user32.dll", "MessageBoxA");//Pointer the the original MBA (MessageBoxA) function.
        DetourAttach(&(PVOID&)originalMBA, (PVOID)HookedMessageBoxA);

        DetourTransactionCommit();
    }

    return TRUE;
}

When I build and run on Debug mode, it crashes on the second MBA call (in the .cpp file of course):
It displays the hooked MBA with "HOOKED" in it like it should, then the it crashes, printing the error below and the program exits with code 3:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.
This is usually a result of calling a function declared with one calling convention with a function
pointer declared with a different calling convention.

I've checked this error and the exit code and found lots of informative posts and solutions, but I couldn't make any of them work for me (maybe I've done something wrong).

NOTE1:
If I'm on Release mode, I can call as many MBA calls as I want as long as they precedes the FreeLibrary call, and all of them will be hooked fine by the .dll file, and the program will exit correctly. But if I try to call the MBA function after the FreeLibrary call - the program crashes with this error:

Exception thrown at 0x50011000 in ProgrammingTask.exe: 0xC0000005: Access violation executing location 0x50011000.

NOTE2:
I tried detaching the dll using DetourDetouch, but it didn't solve it, maybe I did it wrong.
Also, I tried reading about CreateRemoteThread, but was too messy for me.

Thanks in advance.

Your function must be an **exact** replacement for the function you detour. Winapi functions use the `__stdcall` calling convention, yours doesn't. That causes the stack to get imbalanced, lots of grief after it returns. Start fixing it by declaring the function pointer correctly: typedef int(WINAPI * MBA)(HWND, LPCSTR, LPCSTR, UINT); — Hans Passant, Dec 30 '19 at 15:42
@HansPassant I actually removed this before I posted this question to try some different things. Originally, I wrote the function template exactly as you wrote. Still not working. I will edit the original post and add the WINAPI back, thanks for notifying me about this! — Elyasaf755, Dec 30 '19 at 15:51
`if I try to call the MBA function after the FreeLibrary call - the program crashes` - I'm not familiar with Detours, but how is the hook supposed to know that the hook target is now gone? — 500 - Internal Server Error, Dec 30 '19 at 16:30
@500-InternalServerError using DetourDetach and all of its procedure. I didnt add it here because it didn't change a thing. See the code here in the pastebin link, it should be added in the dll: https://pastebin.com/ifc9GcLS But it's still not working — Elyasaf755, Dec 30 '19 at 16:37

Elyasaf755 · Accepted Answer · 2019-12-31T14:15:09.837

1

FOUND IT!

Forgot to add __stdcall to the HookedMessageBox function.

So, instead of

 int HookedMessageBoxA

I rewrote it to this:

 int __stdcall HookedMessageBoxA

Thanks all for your help! <3

edited Dec 31 '19 at 14:15

answered Dec 30 '19 at 19:19

Elyasaf755

2,239
18
24

Hooking with MS Detours crash when hook function is called C++

1 Answers1