Hooking kernel32.dll function stops my program from working

Question

Do you know why I can't run the program when hooking one of kernel32 functions? I'm writing anti cheat and want to optimize it more because currently it's in thread, but something is wrong...

There's written OpenProcess because I've tried before to hook it and the same problem.

typedef HANDLE ( WINAPI * pOpenProcess )(   _In_   HANDLE hProcess,
                                         _In_   LPSECURITY_ATTRIBUTES lpThreadAttributes,
                                         _In_   SIZE_T dwStackSize,
                                         _In_   LPTHREAD_START_ROUTINE lpStartAddress,
                                         _In_   LPVOID lpParameter,
                                         _In_   DWORD dwCreationFlags,
                                         _Out_  LPDWORD lpThreadId );

pOpenProcess original;

__declspec(naked) void hOpenProcess()
{
    __asm PUSHAD
    __asm PUSHFD
        //my actions here
    __asm POPFD
    __asm POPAD
    __asm JMP[original]
};

void ZPerformHook()
{
    DWORD Address = ( DWORD )GetProcAddress( GetModuleHandle( TEXT( "kernel32.dll" ) ), "CreateRemoteThread" );
    original = ( pOpenProcess )DetourFunction( (PBYTE)Address,  (PBYTE)hOpenProcess );
}

The arguments for OpenProcess are passed on the stack. The very first thing you do is altering the stack. That's not good. Your code needs to both preserve the cpu registers *and* not damage the stack. That's non-trivial, best to not write this code yourself. It's been done, you already tagged the question with [detours] so you already know a good solution. There's little point in expecting another one. — Hans Passant, Aug 03 '13 at 15:41
@HansPassant: Whilst I wholeheartedly agree that there is no reason to write your own code to do this, the argument that the stack is being message up above doesn't apply, as the same PUSH operations are matched with POP operations - at least not for the call to the original function. More likely that `//my actions here` are somehow broken (or `original` doesn't contain the correct address, possibly). — Mats Petersson, Aug 03 '13 at 17:41
[Assuming of course that this is compiled for 32-bit code - but I think inline assembler isn't supported in 64-bit mode anyway, and the inline assembler should know that PUSHFD and POPFD and PUSHAD and POPAD don't work for x86-64. — Mats Petersson, Aug 03 '13 at 17:42
@MatsPetersson: the original didn't contain the correct address, problem solved. — deepspace, Aug 03 '13 at 17:43

score 1 · Accepted Answer · answered Aug 03 '13 at 14:09

"//my actions here" would be interesting, maybe you are corrupting the stack. or maybe the error is in your DetourFunction. how does your program fail? maybe with a access violation?

also you don´t have to use a naked function. you can just hook to a function that has the exact same signature as your target. no asm needed.

HANDLE __stdcall hOpenProcess(  HANDLE hProcess,
                                LPSECURITY_ATTRIBUTES lpThreadAttributes,
                                SIZE_T dwStackSize,
                                LPTHREAD_START_ROUTINE lpStartAddress,
                                LPVOID lpParameter,
                                DWORD dwCreationFlags,
                                LPDWORD lpThreadId )
{
    // do your stuff here
    std::cout << "From hook" << std::endl;

    return original( hProcess, lpThreadAttributes, dwStackSize,  lpStartAddress, lpParameter,  dwCreationFlags,  lpThreadId);
}

if that doesn´t work, check the return value of GetProcAddress, if that´s correct, something in your DetourFunction may be going wrong.

you could also use a disassembler like beaengine and dump your target function after detouring to see if the hook was applied correctly

Hooking kernel32.dll function stops my program from working

1 Answers1