Run-Time Check Failure #0 in embedded asm code

Question

I'm a bit new to assembler, but I'm trying to lookup the parameters from a C++ method in the esp stack, using embedded assembler code. So far I haven't even been able to copy the esp pointer to ebp so I can get a hold on the stack (in case it changes). Even this little piece of code gives me the failure:

#include <stdlib.h>

int main(int argc, char* argv[])
{
    __asm
    {
        mov ebp, esp
    }
    system("pause");
    return 0;
}

After I run this, I get:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

Don't know what to do. Help me please, and thanks in advance.

score 2 · Accepted Answer · edited Jan 01 '17 at 02:48

In general you should push the current value of EBP on the stack before you move the value of ESP into EBP. EBP is a "callee-save" register on 32-bit platforms, meaning if you are going to modify it in a function, you must save it first.

If you want your function to return the value of the where the stack is pointing (or where it will return after the function call), the best thing to-do would be the following:

void* get_stack_addr()
{
        void* stack_ptr = NULL;

        //on 32-bit systems
        //EBP is pointing at top of stack frame
        //EBP + 4 is the return instruction address
        //EBP + 8 is the value that was in ESP before function call for a function with no arguments
        __asm__
        (
                "movl %%ebp, %0\n\t"
                "addl $8, %0\n\t"
                : "=r" (stack_ptr)
        );

        return stack_ptr;
}

That way EAX is now holding the address of the value that the stack was pointing to before the call to get_stack_addr() was made. If you just returned the value of ESP in the function, you actually have no idea where you're pointing to, since the compiler often pads the stack in a C/C++ function in order to maintain proper stack alignment. It also often reserves space on the stack for all local variables, which again will throw off the calculation of the stack. By using EBP, which points at the top of the stack frame, you can accurately calculate on a 32-bit platform, the value of the stack before the function call. Finally we place the return value in EAX since on most OS application binary interfaces for C/C++, EAX holds the return value of a function, not EBP.

One more thing ... if you are wanting start of the parameters on the stack for the actual function that calls get_stack_addr(), then change movl %%ebp, %0\n\t to movl (%%ebp), %0)\n\t. That way you're now getting the previous stack-frame base pointer (i.e., the stack frame base pointer of the caller), and by adding a value of +8 to that address, you're getting either the beginning of the parameters stored above the return address, or you're looking at an address pointing into the stack frame for the caller of the current function (i.e., the previous stack frame).

As an enhancement "leal 8(%%ebp), %0\n\t" can replace:

"movl %%ebp, %0\n\t"
"addl $8, %0\n\t"

This leal instruction will add 8 to the value of EBP and store the result in the output operand.

Thanks so much Jason, now I have a better idea on how to implement my solution. Again thanks for your help and time. — Camilo Penagos, May 05 '11 at 19:40
No problem :) ... btw, my in-line assembly is for the GNU toochain, so for Microsoft there are going to be some syntax differences. Also you should reference the Windows ABI, especially for C++ to make sure you're getting the right values for the stack address (i.e., you are making the correct offsets from `EBP` that point to the beginning of the function's arguments, etc.). — Jason, May 05 '11 at 20:03
Using Microsoft's inline assembly syntax, the entire body of the function would simply be `__asm { mov eax, ebp \n add eax, 8 }`. The result can simply be left in `eax`, since all Windows calling conventions use that to return integer-sized values. — Cody Gray - on strike, Dec 30 '16 at 10:22
The GCC inline assembler you have makes a lot of assumptions. Your code really assumes that there is a frame pointer, and that the return value will be passed through the exit of the function via _EAX_. If the compiler chooses to optimize your function by inlining the result may not necessarily end up in _EAX_. you used output constraint `=r` you should change `%%eax` to `%0` to use the actual register assigned by GCC to your template. Technically your code could have been simplified to `leal 8(%%ebp), %0` .If the optimizer eliminates stack frames be aware that `%ebp` may not be a useful value — Michael Petch, Dec 31 '16 at 08:38
Thanks @MichaelPetch for this addendum to my original answer. When it comes to assembly there are a lot various scenarios that could be encountered. In this case I was simply attempting to help the OP with his basic scenario that didn't include inlining, aggressive compiler optimizations, etc. Please feel free to edit my answer with your suggestions and any other improvements you see could benefit other readers. — Jason, Jan 01 '17 at 01:45

Run-Time Check Failure #0 in embedded asm code

1 Answers1

Linked