Why does using a function pointer to call a function bypass the hook?

Question

Using the Microsoft Detours library, I've written the following simple code:

#include <Windows.h>
#include <detours.h>
#include <stdio.h>


void RealFunc(int num) {
    printf("RealFunc %d\n", num);
}

void(*RealFuncPtr)(int) = &RealFunc;

void HookedFunc(int num) {
    printf("HookedFunc %d\n", num + 100);
    // RealFunc(num); // This starts an infinite loop because it calls HookedFunc which calls RealFunc which calls HookedFunc etc...
    (*RealFuncPtr)(num); // This doesn't start an infinite loop and only calls RealFunc without calling HookedFunc. Why is this?
}

int main() {
    RealFunc(100);

    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach((PVOID*) &RealFuncPtr, &HookedFunc); //redirect RealFunc to HookedFunc

    if (DetourTransactionCommit() != NO_ERROR) {
        return 0;
    }

    printf("Hook successful!\n");

    RealFunc(100);

    getchar(); // Pause console
    return 0;
}

Here is the output:

RealFunc 100
Hook successful!
HookedFunc 200
RealFunc 100

As you can see, because of the (*RealFuncPtr)(num); line, the real function is called at the end of the hooked function call. However, if I comment out (*RealFuncPtr)(num); and uncomment RealFunc(num);, it seems to start an infinite loop. Why does the loop only occur when I use RealFunc(num);?

Also, for some reason, when I set the project to release mode, here is the output:

RealFunc 100
Hook successful!
RealFunc 100

The hook seems to not work during release mode. Is it my Visual Studio configuration or something wrong with my code?

score 1 · Accepted Answer · answered May 26 '15 at 16:48

1

That's how Detours work. DetourAttach modifies RealFuncPtr to point to some trampoline code that calls the original function directly, bypassing the hook. That is required for the hook to be able to call the original function, if needed.

As for release mode, the compiler inlines your call to RealFunc, rendering the hook useless. You can either add a level of indirection to bypass this, or apply the no-inline attribute of your compiler to that function.

answered May 26 '15 at 16:48

a553

498
5
12

For anyone struggling to understand, [Figure 1](http://research.microsoft.com/en-us/groups/sn-res/huntusenixnt99.pdf) explains how detours work nicely. – Thomas K. Rice May 29 '15 at 15:36

score 0 · Answer 2 · answered May 26 '15 at 16:14

0

Function hooks are not exactly expected by the compiler or in any standard.

There is obviously a optimization that is bypassing the hook. Turn off various complier optimizations until you find the one that is causing the problem.

Otherwise, it is probably good practice not to use function hooks at all.

answered May 26 '15 at 16:14

Nick Whaley

2,729
2
21
28

Thanks. I can't believe I overlooked this. – Thomas K. Rice May 26 '15 at 16:29

Why does using a function pointer to call a function bypass the hook?

2 Answers2