5

I have got three assembler code fragments and I should identify the proper calling convention (CDECL, STDCALL, FASTCALL) for each fragment. In order to identify the calling convention I was searching for the stack cleaner which is either caller or callee. But what if I cannot find the cleaning code which I assume have look like 'add esp, 8'. Am I wrong with my approach. Must I search for other calling convention characteristics? In the following there are the code fragments.

Fragment 1

push ebp
mov ebp , esp
sub esp , 0x8
mov [ ebp-0x4 ] , eax
mov [ ebp-0x8 ] , edx
mov eax , [ ebp-0x8 ]
mov edx , [ ebp-0x4 ]
add edx , eax
mov eax , ecx
add eax , edx
leave
ret

Fragment 2

push ebp
mov ebp , esp
mov eax , [ ebp+0xC]
mov edx , [ ebp+0x8 ]
add edx , eax
mov eax , [ ebp+0x10 ]
add eax , edx
pop ebp
ret 0xC

Fragment 3

push ebp
mov ebp , esp
mov eax , [ ebp+0xC]
mov edx , [ ebp+0x8 ]
add edx , eax
mov eax , [ ebp+0x10 ]
add eax , edx
pop ebp
ret
Dennis1818
  • 53
  • 1
  • 7
  • wtb highlight for assembler code on stackoverflow :( – Andrew Surzhynskyi May 05 '15 at 19:31
  • `add esp...` is not the only way, [ret also works](http://stackoverflow.com/questions/1396909/ret-retn-retf-how-to-use-them) – Diego May 05 '15 at 19:36
  • so in case of `ret` the return address is popped of the stack and put into `eip`. So its up to the caller to clean up the stack, right? – Dennis1818 May 05 '15 at 19:48
  • With the stdcall and fastcall calling conventions it's the function's job to pop arguments from the stack. One of your functions doesn't have any arguments on the stack, one of them pop arguments from the stack and one of them doesn't pop arguments from the stack. – Ross Ridge May 05 '15 at 20:54

2 Answers2

4

first Fragment is fastcall calling convention because the procedure has used the registers(EAX,EDX) without assigning values to it - which means the caller used the registers for passing the arguments - like in line

mov [ ebp-0x4 ] , eax

second Fragment is stdcall calling convention because the procedure cleaned the stack off the parameters

ret 0xC

third Fragment is cdecl calling convention because the procedure took the parameters from the stack but it didn't cleared the stack off the parameters

ret

quick marks for each one will be:

fastcall : the caller use registers for passing first two arguments.

stdcall : the callee has to clean the stack.

cdecl : the caller has to clean the stack.

for more information see the wiki

http://en.wikipedia.org/wiki/X86_calling_conventions

the accountant
  • 506
  • 6
  • 14
2

Since the return address is in the way, the typical method for callee cleanup is the ret X instruction (where X is the number of argument bytes to remove). As such, your second example is callee cleanup, ie. stdcall.

To spot fastcall, you just have to find registers that are used without being initialized. You can see this in your first fragment: neither eax nor edx has been initialized. So that's fastcall.

What remains must be cdecl, but of course you can see that it is accessing arguments above the return address, via ebp+0x8 and ebp+0xC, and it doesn't remove them.

Jester
  • 56,577
  • 4
  • 81
  • 125