Hooking (hot patching) class member functions. Modifying vtable entries

Question

I have 2 questions, after a rather long preamble.

By looking at any function pointer as at a void* I am able to modify its first instructions, transform them into a jmp (either 32 bit relative or 64 bit absolute, via r11, depending on x86/x86-64). I believe that looking at function code as at data is illegal in both C and C++, but it somehow seems to work, in an unsupported way, in both MSVC (Win32) and GCC (OS X). There are several places on the Internet saying that casting function pointers to void* is illegal.

One doesn't simply get a pointer to a class member. I mean, the compiler directly throws errors at build time when trying to look at such a pointer in the same way as I'd look at a void *, practice which seems to work just fine for non-member functions.

Fortunately, in order to hook Direct3D9, I'm working with stuff like IDirect3DDevice9 which has a vtable. With pDev of type IDirect3DDevice9*, it's enough that I look at pDev as at a PVOID*. Then, the first value at pDev is the address of an array of pointers to functions (the vtable):

// IDirect3DDevice9::Present()
typedef HRESULT (CALLBACK *PRESENT_PROC)(
    LPDIRECT3DDEVICE9, const RECT*,
    const RECT*,
    HWND,
    const RGNDATA*
);

PVOID (*vPtr)[] = reinterpret_cast<PVOID (*)[]>(
    *reinterpret_cast<PVOID*>(pDev)
);
PRESENT_PROC pDevicePresent = reinterpret_cast<PRESENT_PROC>(
    (*vPtr)[17]
);

since Present is the 18th entry.

The first answer from here gives a more elegant, higher-level method, starting with defining CINTERFACE. I haven't tested it yet, but according to it I can do things like

reinterpret_cast<PVOID>(pDev->lpVtbl->Present)

without error.

First problem. I'm not an awesome C++ programmer; how do I get a pointer, in general, to a member function, so that I can overwrite the executable bytes of that function. For a non-member I do:

#include <windows.h>
#include <cstdio>
using namespace std;

const unsigned char OP_JMP = 0xE9;  // 32 bit relative jmp
const SIZE_T SIZE_PATCH = 5;        // jmp dword ptr distance; 1 byte + 4 bytes
typedef void (*MyProc)();

void SimpleFunction1()
{
    printf("foo\n");
}

void SimpleFunction2()
{
    printf("bar\n");
}

int main()
{
    PBYTE foo = reinterpret_cast<PBYTE>(SimpleFunction1);
    PBYTE bar = reinterpret_cast<PBYTE>(SimpleFunction2);

    DWORD oldProtection;
    // make sure the bytes of the function are writable
    // by default they are only readable and executable
    BOOL res = VirtualProtect(
        foo,
        SIZE_PATCH,
        PAGE_EXECUTE_READWRITE,
        &oldProtection
    );
    if (!res) return 1;

    // be mindful of pointer arithmetic
    // works with PBYTE, won't with PDWORD
    DWORD distanceToNewFoo = bar - foo - SIZE_PATCH;

    *foo = OP_JMP;
    *reinterpret_cast<PDWORD>(foo + 1) = distanceToNewFoo;

    // called though the pointer instead of foo()
    // to make sure the compiler won't inline or do some other stupid stuff
    reinterpret_cast<MyProc>(foo)(); // will print "bar\n"
    return 0;
}

and something in the same vein for x86-64. For a virtual member of an object, I get the foo pointer from the vtable itself as I've shown above:

reinterpret_cast<FUNC_TYPE>(
    *(reinterpret_cast<void**>(
        *reinterpret_cast<void**>(objptr)) + n
    )
)

Second problem. Can't I get by with simply modifying the entries from the vtable for my object? Here's an example, needless to say, it doesn't work for the pDev object, as taken directly from Direct3D, but Taksi seems to use this method:

#include <cstdio>
using namespace std;

class BaseClass
{
public:
    BaseClass(int a = 0, int b = 0);
    int GetA();
    int GetB();
    virtual void Test();
private:
    int _a;
    int _b;
};

BaseClass::BaseClass(int a, int b) :
    _a(a),
    _b(b)
{
}

int BaseClass::GetA()
{
    return _a;
}

int BaseClass::GetB()
{
    return _b;
}

void BaseClass::Test()
{
    printf("test %d; %d\n", _a, _b);
}

void TheNewFunction(BaseClass *bc)
{
    printf("I am an intruder\n");
}

typedef void (*PROC_TYPE)(BaseClass *);

int main()
{
    BaseClass foo(5, 56);
    PROC_TYPE proc = 0;
    proc = reinterpret_cast<PROC_TYPE>(
        *reinterpret_cast<void**>(
            *reinterpret_cast<void**>(&foo)
        )
    );
    proc(&foo);
    reinterpret_cast<void**>(
        *reinterpret_cast<void**>(&foo)
    )[0] = reinterpret_cast<void*>(TheNewFunction);

    foo.Test(); // runs same old Test(); maybe due to compiler optimization?
    proc = reinterpret_cast<PROC_TYPE>(
        *reinterpret_cast<void**>(
            *reinterpret_cast<void**>(&foo)
        )
    );

    proc(&foo); // runs TheNewFunction
    BaseClass *goo = &foo;
    goo->Test(); // runs TheNewFunction
    return 0;
}

Answer 1

The fastest way to perform this ugly cast (member function to void *), would be the infamous union_cast<>:

template <class T1, class T2>
T1 union_cast(T2 v)
{
  static_assert(sizeof(T1) >= sizeof(T2), "Bad union_cast!");
  union UT {T1 t1; T2 t2;} u {};
  u.t2 = v;
  return u.t1;
}

Use like this:

class MyClass
{
public:
  void foo(int);
};

auto p = union_cast<void *>(&MyClass::foo);

Now, I've given you a loaded gun with the safety off. Please use with care...

Answer 2

As you pointed-out, your method is not "portable", but it actually works in your specific case. And there's no problem IMHO.

First answer:

This is the syntax for working with member function pointers:

class SomeClass {
    int SomeFunc(int, int);
};

int (SomeClass::* pfn)(int, int); // variable pfn is a pointer to a SomeClass's member function

pfn = &SomeClass::SomeFunc; // assign this variable to the member function with the adequate prototype.

SomeClass obj; // instance of this class

int res = (obj.*pfn)(1, 2); // call the member function pointer

Second answer:

You may modify the members of the vtable directly, but you should know that by this you subclass all the objects of this class (and possibly some derived classes).

If you want to subclass only one specific obj you should create another function table, and overwrite the object's vtable to point to the new function table.

Answer 3

class Original {
public:
    int a;

    Original* open(const char *filename, int openmode) {
        printf("Original: %s and %d // %d\n", filename, openmode, this->a);
        this->a = openmode;
        return this;
    }
};

class Group {
public:
    Original* my_open(const char *filename, int openmode) {
        Original *self = (Original*) this;
        printf("Fake:: %s and %d // %d\n", filename, openmode, self->a);
        return self;
    }
};

#define GetPointerToClassMethod(RETURN, CLASS, METHOD, ...) __GetPointerToClassMethod<CLASS, RETURN (CLASS::*)(__VA_ARGS__)>(&CLASS::METHOD)

template <class _Class, class _MethodPrototype>
LPVOID __GetPointerToClassMethod(_MethodPrototype method) {
    return *(LPVOID*) &method;
}

int main() {
    Original o;
    o.open("Dorian", 15);

    DirectHookProcedure(
        GetPointerToClassMethod(Original*, Original, open, const char*, int),
        GetPointerToClassMethod(Original*, Group, my_open, const char*, int)
    );

    o.open("Langbeck", 20);
    return 0;
}

Answer 4

There is no portable way to do this in C++ as the language specification says that any sort of casting you might do from a pointer-to-member-function or pointer-to-function to a void* results in undefined behavior. If you want to do this sort of dynamic code rewriting, you'll need to consult the compiler docs for the particular platform you're working on.

If you are using a compiler that generates vtables you should be able to change where member function code is located by blasting the vtable, assuming that the vtable is mutable. I believe most compilers drop the vtable in a read-only segment so you can't d this accidentally (or maliciously), though. You would also have to worry about compiler optimizations to inline the call, since there's nothing stopping the compiler from recognizing the target of a method invocation in some cases and just hardcoding the call.

Answer 5

You are certainly correct that in some cases, like this one

BaseClass foo(5, 56);
//...
foo.Test(); // runs same old Test(); maybe due to compiler optimization?

the compiler can easily see the type of foo and doesn't need to use a vtable at all. It knows what function to call anyway.

When you call through a pointer to the object, the compiler designers haven't bothered to check if it is always the same type. In your code they surely could see that, but perhaps such an optimization is not useful enough in real code? Or they might add it in the next release of the compiler. :-)

And we should not forget that vtables, their existence and possible layout are all implementation details that the language standard says nothing about.

Answer 6

Almost portable solution to the first problem - va_list in a helper function:

void *DisMember(size_t size, ...)
{
    if (size != sizeof(void *)) return NULL;
    va_list args;
    va_start(args, size);
    void *res = va_arg(args, void *);
    va_end(args);
    return res;
}

// snip
void Base::MyMethod() { /* ... */ }
// snip

void *anything = DisMember(sizeof(void (Base::*)()), &Base::MyMethod);

valdo nailed the second one.