Calling convention is wrong and I can't figure out. C++

Question

I'm trying to detour a function in an application. I have it's source available.

class Foot : public Something, public SomethingElse {    
public:
    Foot( double SomethingAgain, double Somethings,
                      Blahblah *AnotherThing = NULL );

    virtual ~Foot();
    virtual void detourmepls( const char *ARGUMENT );
};

This is how it was defined in the header file.

This is what I used to define it so I would call it from my dll using the correct address.

void (__cdecl* detourmepls)(const char *stuff);

What's the true calling convention for this virtual void function?

Any help is appreciated.

NEW EDITS AND INFOS:

I'm using MS Detours 1.5

detourmepls = (void(__thiscall*)(void *Pthis,void *Unknown,const char *))DetourFunction((PBYTE)ADDRESS, (PBYTE)hookedFunction);

This is how I tried the detour ( its pretty messed up. )

I assume you are passing the function as a C-callback, which is not possible. If registering the callback function allows passing user data, write a static function and pass that with the 'this' pointer to the registering functions. — , Apr 18 '14 at 10:42
application crashes when I attempt to call the original function. — user3548518, Apr 18 '14 at 10:43
Do you use microsoft detours? If so, this could be related since you are trying to deour a non-static member function: http://stackoverflow.com/questions/11048176/detouring-a-member-function-via-an-injected-dll BTW you could get more help, if you posted more code (how you hook the function and how you try to call it, etc...) — Hayri Uğur Koltuk, Apr 18 '14 at 10:49
possible duplicate of [Hooking/Detour Virtual Functions](http://stackoverflow.com/questions/7376897/hooking-detour-virtual-functions) — Hans Passant, Apr 18 '14 at 11:37

score 0 · Answer 1 · answered Apr 18 '14 at 10:55

Background

Callbacks with virtual functions require a 'this' object to resolve the virtual function. A virtual function pointer is actually more like a vtable "offset" (and quite hairy in the presence of multiple virtual double-secret inheritance ;).

__cdecl doesn't support any of that.

What does the API were you register your callback look like? It sound's like a C-api. These sometimes take a void*. If not, then you need to store the object somehwere else.

Callback is function only, no data

If it's only a callback function, and no user data, you need to store the object pointer seprately on your side and resolve the virtual function call in a static function, which is the one you register.

void registerCallback(void (__cdecl* callbackfn)(const char* stuff));

class Foo {
     static Foo* s_callback_object;

     ~Foo() { 
          // never hurts
          if (this == s_callback_object) 
              s_callback_object = 0; 
     } 

     static void Foo::callbackResolver(const char* stuff) {
          assert(s_callback_object);
          s_callback_object->detourmepls(stuff);     
     }
}

Fop* foo::s_callback_object = 0;

Foo myFoo;
Foo::s_callback_object = &myFoo;
registerCallback(&Foo::callBackResolver);

Callback function with user data arg:

(This is how it should be done, folks)

For case 2, user data, you can cast that to your object instead:

void registerCallback(void (__cdecl* callbackfn)(const char* stuff),
                     void *userdata);

class Foo {
     static void Foo::callbackResolver(const char* stuff, void *userdata) {
          Foo* foo = reinterpret_cast<Foo*>(userdata);
          foo->detourmepls(stuff);     
     }
}

Foo myFoo;
registerCallBack(&Foo::callbackResolver, &myFoo);

The latter scheme is immensely useful even in plain C programming, so if you do your own API's, do it like this. :)

I'm using MS Detours 1.5 – user3548518 Apr 18 '14 at 14:32 — user3548518, Apr 18 '14 at 14:32

Calling convention is wrong and I can't figure out. C++

1 Answers1