Calling a function that can be either cdecl or stdcall

Question

I need to write code that calls an external function that can be either stdcall call or cdecl in a 32bit windows application.
My code, the caller, can't know in advance which of these its going to be. Right now, if I try to call a cdecl function from a call site that was defined as stdcall, I get a checkEsp exception dialog, and I'm guessing that's there for a good reason.
Is there any way to do it?

Try using a foreign function interface (FFI) library. – Kerrek SB Jul 13 '13 at 00:31 — Kerrek SB, Jul 13 '13 at 00:31
FFI still needs to know the calling convention – shoosh Jul 13 '13 at 00:45 — shoosh, Jul 13 '13 at 00:45
@HansPassant the arguments order is the same – shoosh Jul 13 '13 at 14:09 — shoosh, Jul 13 '13 at 14:09

score 4 · Accepted Answer · edited Feb 23 '21 at 04:15

It can be done following way:

          mov     esi, esp

          push    arg3
          push    arg2
          push    arg1
          call    [SomeExternalProc]

          mov     esp, esi   ; now the stack is always properly cleaned

The external procedure will preserve esi. Or you can use any other register preserved by the external procedure or even memory variable - local or global.

Good, the order of the arguments is the same for CDECL and STDCALL - in reverse order. (Left-most arg at the lowest address.) So they're compatible except for where ESP points on return. Both conventions agree on which registers are call-preserved vs. call-clobbered.

score 1 · Answer 2 · answered May 09 '14 at 21:25

1

You can also use alloca() which has the side effect of saving and restoring the stack pointer:

{
    alloca( (uintptr_t)callback & 2 );
    callback();
}

answered May 09 '14 at 21:25

Sam Lantinga

141
1
5

score 0 · Answer 3 · answered Jul 13 '13 at 00:31

0

cdecl and stdcall are by definition incompatible. In cdecl, the caller cleans up the stack, in stdcall the callee cleans up the stack. If you assume stdcall, but it is in fact cdecl, no one cleans up the stack. That means your ESP (stack pointer) is going to be screwed up after the call. Maybe if you give more details, there maybe a work around, but there is no way to call a function without knowing it's calling convention without messing up your stack.

See : http://en.wikipedia.org/wiki/X86_calling_conventions for a definition of the difference.

answered Jul 13 '13 at 00:31

Enkid

11

Well, is there a reliable way to check this situation and just fix ESP if that's needed? What does checkEsp do anyway? – shoosh Jul 13 '13 at 00:44
ESP is your stack pointer. It keeps track of where your stack is currently located. Theoretically, you might be able to save ESP to a variable on the heap and retrieve it after the call, but that would be very messy... and I've never tried that, so I'm not even sure that would work. I'm not sure of the mechanism check ESP uses to make sure ESP is properly maintained before and after the call. A better strategy would be to check the calling convention and adjusting your call based on that, though I haven't seen a consistent method to programatically check the calling convention at runtime. – Enkid Jul 13 '13 at 01:41

Calling a function that can be either cdecl or stdcall

3 Answers3

Linked