8

In VS2010, the managed debugging assistant will give you a pInvokeStackImbalance exception (pInvokeStackImbalance MDA) if you call a function using the wrong calling convention, commonly because you didn't specify CallingConvention = Cdecl when calling a C library. E.g. you wrote

[DllImport("some_c_lib.dll")]
static extern void my_c_function(int arg1, int arg2);

instead of

[DllImport("some_c_lib.dll", CallingConvention=CallingConvention.Cdecl)]
static extern void my_c_function(int arg1, int arg2);

and thus got the StdCall calling convention instead of Cdelc.

If you answer this, you already know the difference, but for other visitors to this thread: StdCall means that the callee cleans the arguments from the stack, whereas Cdecl means that the caller cleans up the stack.

So, if you get the calling convention wrong in your C code, your stack doesn't get cleaned up and your program crashes.

However, .NET programs don't seem to crash even though they use StdCall for Cdecl functions. The stack imbalance check wasn't enabled by default on VS2008, so some VS2008 projects use the wrong calling convention unbeknownst to their authors. I just tried GnuMpDotNet and the sample runs just fine even though the Cdelc declaration is missing. The same holds true for X-MPIR.

They both throw the pInvokeStackImbalance MDA exception in debug mode, but don't crash in release mode. Why is this? Does the .NET VM wrap all calls to native code and restore the stack itself afterwards? If so, why bother with the CallingConvention property at all?

John Reynolds
  • 4,927
  • 4
  • 34
  • 42
  • Just found out that the calling convention for 64-bit Windows is unified. It's Cdecl all the way, which explains why there's no pInvokeStackImbalance MDA exception on x64. See Raymond Chen, http://blogs.msdn.com/b/oldnewthing/archive/2004/01/14/58579.aspx – John Reynolds Mar 19 '11 at 09:33

2 Answers2

7

It is because of the way the stack pointer is restored when the method exits. The standard prologue of a method, shown for the x86 jitter;

00000000  push        ebp                 ; save old base pointer
00000001  mov         ebp,esp             ; setup base pointer to point to activation frame
00000003  sub         esp,34h             ; reserve space for local variables

And the way it ends:

0000014a  mov         esp,ebp             ; restore stack pointer
0000014c  pop         ebp                 ; restore base pointer
0000014d  ret

Getting the esp value unbalanced is not a problem here, it gets restored from the ebp register value. However, the jitter optimizer not infrequently optimizes this away when it can store local variables in cpu registers. You'll crash and burn when the RET instruction retrieves the wrong return address from the stack. Hopefully anyway, really nasty when it happens to land on a chunk of machine code by accident.

This is liable to happen when you run the release build without a debugger, tough to troubleshoot if you didn't have the MDA to help you.

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • So what happens when I call native code? The JIT doesn't do anything with the callee then. Does it generate a delegate with the same prologue/epilogue (except for the local variable allocation, of course) and the call to native code in between? – John Reynolds Mar 18 '11 at 22:38
  • You don't call native code directly, you call the pinvoke marshaller. It does whatever it needs to do to setup the unmanaged call stack, directed by your [DllImport] declaration before making the call. How exactly it returns with the stack out of balance isn't obvious, the marshaller's source code isn't readily available. Clearly it survives it. – Hans Passant Mar 18 '11 at 22:53
  • When I get the time, I'll test this on Mono. If it works there too, the Mono source code for Marshal.GetDelegateForFunctionPointer could give some insights of how it might work, even if .NET is a very different animal than Mono internally. – John Reynolds Mar 18 '11 at 23:14
3

The runtime can detect the stack imbalance because the stack pointer isn't where it's expected. That is, in the case of StdCall, where the called function is expected to clean up the stack, then the runtime could do this: 

SavedSP = SP; // save the stack pointer
// now push parameters
// call the external function.
if (SP != SavedSP)
{
    // error!
}

Now, if the value of SP is less than SavedSP, then there's extra stuff on the stack--meaning that the runtime can just go ahead and restore the saved stack pointer.

The runtime should always be able to detect a stack imbalance. Whether or not it can always recover is unknown to me. But in the case of inadvertently calling a Cdecl method as StdCall, it should be able to recover without trouble, since there will be extra stuff on the stack that it can ignore.

As to why bother? As you say, the difference between StdCall and Cdecl is really only who's responsible for stack cleanup. Also, StdCall is not compatible with variable argument lists (i.e. printf in C), although I don't know if it's even possible to call such a method from .NET (haven't ever had a need to). In any case, although there doesn't appear to be a particular problem with calling a Cdecl method with StdCall, I kind of like knowing that there's a potential error. To me, it's like the error message that the compiler gives when you write:

uint x = 3;
int y = x;  // error!

I know that the assignment is okay, but the compiler disallows it because it's a potential source of bugs. In my mind, an unbalanced stack is a potential source of bugs. No, it is a bug that can cause some very bad things to happen. I'd rather the runtime told me about it so that I can fix the problem.

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351
  • And where is SavedSP stored? (on the stack? but SP is wrong! thread-local variable? that might be more expensive than MS probably wants, and consider the fact that native code can call back into managed code, which can call back to native code again.) – Qwertie Mar 18 '11 at 23:32
  • Qwertie: The code could store the stack pointer in BX or any other register that the callee is supposed to preserve. Come to think of it, the runtime could compute the expected value of SP based on BP (which is supposed to be saved). The calling conventions all specify which registers should be preserved. I'm not saying that the runtime can *always* recover from a corrupted stack. Obviously, an overwritten stack is unrecoverable. But in the case of calling a `Cdecl` function with `StdCall` semantics or other situations where stuff is left on the stack, recovery is very easy. – Jim Mischel Mar 19 '11 at 14:29