how C compiler treats a struct return value from a function, in ASM

Question

When speaking about C function's return value, the return value is stored in the EAX register. Suppose we are speaking about 32 bit register, integers are welcomed, but what happens when we return these types: long long, long double, a struct/union that is larger than 32bit.

For `struct` return types, the typical solution is for the *caller* to provide space for the object and, say, pass a pointer to that space in EBX, or have a convention that this space is at the end of the stack before the function call, etc. The callee then writes the return value into that space. — Kerrek SB, Jul 14 '14 at 16:30
If size of return value is not small (greater than 4 bytes 32bit machine, for example) then compiler can add "extra" parameter to argument list and return value by pointer. Or using stack, it depends on implementation. — Ivan Ivanov, Jul 14 '14 at 16:33
I didn't quite get it: Kerrek: the caller, after pushing the parameters into the stack, pushes another space according to the type of the return value? — Medals, Jul 14 '14 at 18:03
@Medals: The caller allocates a buffer in the calling scope and passes a pointer to said buffer as a parameter to the callee. — 500 - Internal Server Error, Jul 14 '14 at 19:44
Where's the problem ? It returns the struct "by content" but by basically it's just an address, a pointer to a struct created with all of the variables in it, and no problem returning a pointer which is the size of an int — Zach P, Jul 14 '14 at 20:05
the return value is stored in eax or edx:eax in x86 and rax or rdx:rax in x86_64, not ebx — phuclv, Jul 15 '14 at 01:05

Peter Cordes · Answer 1 · 2022-01-19T03:09:20.150

In common x86 calling conventions, objects that fit in two registers are returned in RDX:RAX. This is the same register pair that is an implicit input/output for div and mul instructions, and for cdq / cqo (sign extend e/rax into e/rdx).

The i386 Linux (SysV) calling convention only returns 64bit integers that way. Structs (even a struct consisting of a single int32_t) use the hidden-parameter method instead of being packed eax or edx:eax. 64bit Linux, and Microsoft's current standard __vectorcall, both pack structs into e/rax, or e/rdx:e/rax.

Many calling conventions handle larger objects by adding a hidden extra parameter: a pointer to space for storing the return value. Consult the ABI documentation for the specific ABI you are using. (links at in the x86 wiki).

Compared to other calling conventions discussed in comments (e.g. implicitly using space on the stack to store large objects being returned), passing a pointer can save a copy, because the pointer can point to the final destination instead of scratch space on the stack.

But usually only if the final destination is on the stack. The callee can assume the return-value object is not the same object as / doesn't alias any global or any memory it can reach via pointers. i.e. the caller has to do escape analysis. See What prevents the usage of a function argument as hidden pointer? - In the C abstract machine, the return value object isn't written until the moment the function is returning, after it's done reading anything.

But it can point to whatever convenient place in the caller's stack frame, not a fixed place relative to RSP, which may avoid a copy.

score 6 · Answer 2 · 2014-07-15T08:09:30.857

Consider this program:

struct object_t
{
  int m1;
  int m2;
  int m3;
};

struct object_t
test1(void)
{
  struct object_t o = {1, 2, 3};
  return o;
}

long long
test2(void)
{
  return 0LL;
}

long double
test3(void)
{
  return 0.0L;
}

compiled on Windows with (object file, minimum instructions, with no x87 instructions):

$ gcc -Wall -c -O2 -mno-80387 test.c -o test.o

The first function:

00000000 <_test1>:
   0:   8b 44 24 04             mov    eax,DWORD PTR [esp+0x4]
   4:   c7 00 01 00 00 00       mov    DWORD PTR [eax],0x1
   a:   c7 40 04 02 00 00 00    mov    DWORD PTR [eax+0x4],0x2
  11:   c7 40 08 03 00 00 00    mov    DWORD PTR [eax+0x8],0x3
  18:   c3                      ret

The caller will provide a pointer to where his structure is over the stack as first argument and test1 will fill it using that pointer.

Second function (sizeof(long long) == 8):

00000020 <_test2>:
  20:   31 c0                   xor    eax,eax
  22:   31 d2                   xor    edx,edx
  24:   c3                      ret

The result will be returned over two registers eax and edx, not just eax.

Third function (sizeof(long double) == 12):

00000030 <_test3>:
  30:   31 c0                   xor    eax,eax
  32:   31 d2                   xor    edx,edx
  34:   31 c9                   xor    ecx,ecx
  36:   c3                      ret

The return value will be passed over three registers, eax, edx, ecx.

how C compiler treats a struct return value from a function, in ASM

2 Answers2

Linked

Related