Is my stdcall wrapper for usercall function correct?

Question

I need to wrap the below __usercall function to _cdecl/_stdcall:

char __usercall sub_4017B0<al>(int a1<ebx>, int a2)

a1 is integer, a2 is actually an arry of ints ('int args[10]')

Is this correct? What does the <al> mean behind sub_4017B0 ?

int __stdcall func_hook_payload(int callnum, int* args);

// Wrapper for
// char __usercall sub_4017B0<al>(int callnum<ebx>, int a2)
__declspec(naked) void func_hook()
{__asm{
    push ebp
    mov ebp, esp

    push dword ptr[ebp + 0x28] // args[9]
    push dword ptr[ebp + 0x24] // args[8]
    push dword ptr[ebp + 0x20] // args[7]
    push dword ptr[ebp + 0x1C] // args[6]
    push dword ptr[ebp + 0x18] // args[5]
    push dword ptr[ebp + 0x14] // args[4]
    push dword ptr[ebp + 0x10] // args[3]
    push dword ptr[ebp + 0x0C] // args[2]
    push dword ptr[ebp + 0x08] // args[1]
    push dword ptr[ebp + 0x04] // args[0]
    push ebx // callnum
    call func_hook_payload
    leave
    ret // note: __usercall is cdecl-like
}}

How would a wrapper look like for calling sub_4017B0 ?
The wrapper should have this signature:

int sub_4017B0_wrapper(int callnum, int* args);

you'd better specify compiler, version, platform, all those details because none of this is standard. — Ben Voigt, Jan 28 '11 at 00:49
Windows 7 32 Bit, MinGW g++ (latest stable), or Visual C++ 2008 — Stefan Steiger, Jan 28 '11 at 00:51
Why would you write the wrapper in assembly instead of in C? You could just write a new function with a proper signature that calls the function with the "wrong" convention. — sharptooth, Jan 28 '11 at 06:33
@sharptooth: That's exacly the problem. __usercall is not a calling convention but a compiler optimization. There's no proper calling convention like __stdcall or __fastcall to do this in C/C++. However, if you can prove me wrong, I'd very much appreciate it. — Stefan Steiger, Jan 28 '11 at 07:35
@Quandary: Is that `__usercall` function declared anywhere? If it is it means that the compiler can call it somehow - and it doesn't matter if `__usercall` is a convention or not - it can just call it with some magic. Now if you introduce a wrapper with `__stdcall` convention and insert a call to that function into the wrapper why will that function not be called? — sharptooth, Jan 28 '11 at 07:41
It's defined in a source that I don't have. I have reverse-engineered it, so I have the address with which I can make a functionpointer (and the pseudo-decompile source above), nothing more. See http://stackoverflow.com/questions/4823401/hooking-usercall-function, I've more details there. — Stefan Steiger, Jan 28 '11 at 09:59

score 3 · Accepted Answer · edited Feb 14 '13 at 04:18

does the function take an actual int* or does it take va_args? in cases like this you need to provide the original calling code to.

from what I can gather, your wrapper should look like this(I don't use stack frames, but your frame is wrong as you don't pop ebp before returning):

__declspec(naked) void func_hook()
{
    __asm
    {
        push dword [esp + 4]    //int* - pArgs
        push ebx                //int - nArgs
        call func_hook_payload  //you can even just jump to this, the stack should clean itself up correctly
        retn
    }
}

should it be va_args you can do something like this:

__declspec(naked) void func_hook()
{
    __asm
    {
        lea eax,[esp + 4]       //int* - &nArg[0]: here we abuse the way the windows stack grows, creating a stack based buffer 
        push eax                //int* - pArgs
        push ebx                //int - nArgs
        call func_hook_payload
        retn
    }
}

Calling the old func is pretty simple too, you can do it without a nake function, but really I prefer naked funcs :)

void __declspec(naked) __stdcall CallTheOldVMFunc(int nArgs, int* pArgs)
{
    __asm
    {
        push ebx                //save ebx, its not a scratch register
        mov ebx,[esp + 8]       //set the number of args
        push [esp + 12]         //push the arg ptr
        call TheOldVMFunc
        pop ebx                 //restore ebx
        retn 8                  //ret and cleanup
    }
}

Actually, the C code is: intptr_t CL_CgameSystemCalls( intptr_t *args ) {...}, and IMHO, it's not really clear whether it's va_arg or int*. I will just have to try. — Stefan Steiger, Jan 28 '11 at 07:41
Thank you very much. Can you also provide me with a wrapper for calling this function (this is the wrapper to 'receive' the function). — Stefan Steiger, Jan 28 '11 at 07:47
@Quandary: sure, seeing as its simple. knowing quake, i'm pretty sure its `int*`, as they have a conversion from the `va_args` to `int*` in the vm code. — Necrolis, Jan 28 '11 at 08:41
Gr8, many thanks. Whether it works or not I'll find out this evening (CET). PS: Actually it is UrbanTerror's new 4.2 executable (www.urbanterror.net) — Stefan Steiger, Jan 28 '11 at 09:45
Tested, works ! Many thanks, great help ! I'd like to give you 1000 upvotes, but I can only give you one. — Stefan Steiger, Jan 28 '11 at 18:26

Is my stdcall wrapper for usercall function correct?

1 Answers1