I have been doing some research into inline assembly in C and how the call stack works but I have been unable to figure out if it's at all possible to retrieve the address of a variable that is requesting the return value of a function, from WITHIN the function.
int hypothetical_func(){
/*...
.. some assembly to get the address of 'int a' from the call stack?
...*/
return 5;
}
int main(){
int a = hypothetical_func();
}
Is this at all possible?
NO. int is returned in a register, and the callee has no involvement in what the caller does with that register after it returns. It might never be stored in memory.
If the return-type wasn't int, but instead something large enough that the calling convention returned it by value, then hypothetical_func would have an address for an output. (Or a hypothetical (terrible) calling convention might return even int via hidden pointer instead of a register. Assuming the machine is a register machine like all real CPUs.)
But that might just be a return-value temporary, not the actual LHS of an assignment. (Or initialization, which is close enough to the same thing in C, if not C++). Especially if the assignment is to a global or something. See What prevents the usage of a function argument as hidden pointer? for the case of *out = foo(); where T *out is a function arg. It's highly non-trivial to prove if/when it's safe to pass that function arg along as the return-value object for foo().
And some compilers don't even try to optimize, and just make space on the stack for the return-value temporary and copy from there into the final object.
And as @prl points out, the return-value might not even be the initializer for a variable. e.g. printf("%d\n", foo()); just passes on the return value to a function arg. Or foo(); discards the return value, not assigning it anywhere. (But if the calling convention specifies that the function returns by hidden pointer, the caller must pass a pointer to enough scratch space. The callee is still going to write its return value and needs to not segfault from a bad pointer or overwrite something else. That's an asm / calling-convention detail separate from the operation of the C abstract machine. Or I guess you could say the return-value object still exists, it's just not assigned anywhere.)
Plus with inline assembly, you don't even have access to that. Unless you count writing a __attribute__((naked)) function where you still write the whole function inside an asm statement, and the compiler doesn't handle anything except the name-mangling of the function name. No prologue or epilogue, or abstracting away the calling convention with C variables for args and one that you return. (/grumble that C compilers can't create functions that return multiple separate values in multiple registers like you can in hand-written asm.)
But even with hand-written asm, there's no way to do this for normal calling conventions on normal ISAs like x86 and ARM. The return-value object for int is just a register.
It’s not possible in any practical sense. The return value may not be assigned to a variable, and if it is, the variable may not have a memory address. Even if neither of those is the case, and the return value is assigned to a variable in memory, the address of that variable isn’t available to the function.
An impractical way to figure this out (if needed for debugging, for example) is to get the return address from the stack and disassemble instructions looking for one that writes the return value register to memory. Such an instruction, if it exists, will typically be within a few instructions after the return point.
If int a = hypothetical_func(); is inside another function, it should be on the stackframe of that function, thus using backtrace() you could find that functions stackframe and locate that variable.
For more extraordinary information: https://www.linuxjournal.com/article/6391
