dynamic library function cast

Question

A newbie question about shared library: In C, when loading a dynamic library, we use dlopen and then dlsym to find a symbol or a function. Now let say, the function we look for in the dll is typed as:

int add(int a, int b);

but if we cast it to another type, say,

typedef int (*sum)(int a, int b, int c);

what will happen? Will C runtime complain about it?

Thanks!

No, the C runtime will not complain. But invoking the function pointer may not work (that is, the result is undefined). — jxh, Apr 26 '14 at 00:31
@Guoqin That's exactly what "UB" means. Anything could happen - from "it works flawlessly" to "my computer went up in smoke". In practice, on x86-64 under Linux using the 64-bit SYSV ABI, nothing is likely to happen; here an extra integer argument will be placed in register `rdx`, which the function will simply ignore. I would not however be so sure about other ABIs. — Iwillnotexist Idonotexist, Apr 26 '14 at 00:47
@iwillnotexist Got you! A side question I thought most platform pass the parameter using stack rather than register, right? — Alfred, Apr 26 '14 at 01:02
@Guoqin: Depends on ABI. In x86 `stdcall`, for example, all arguments are pushed on the stack, and the callee restores the stack. If the function was `stdcall`, then, calling it with the wrong number of arguments will imbalance the stack. That is very bad news. — nneonneo, Apr 26 '14 at 01:03
But with other ABIs, like the 64-bit ABI mentioned by @IwillnotexistIdonotexist, or the ARM ABI, a certain number of arguments are placed in registers first, and excess arguments go on the stack. — nneonneo, Apr 26 '14 at 01:04

score 2 · Accepted Answer · answered Apr 26 '14 at 01:08

Long story short, this is undefined behaviour. Calling a function with the wrong number of parameters in C (which you can do by casting function pointers even without dlopen/dlfree) produces undefined behaviour.

For callee-clean calling conventions, like stdcall, using the wrong number or type of arguments will produce a stack imbalance (the callee adjusts the stack to the wrong place). A stack imbalance will quickly kill the program (if you're lucky) by corrupting the return address and local variables of the caller.

For caller-clean calling conventions, the effect is much like calling e.g. printf with the wrong number of arguments: the function may behave erratically by using garbage arguments, but your program might not blow up. (This in general is not desirable; an attacker could exploit this to take control of your program, for example).

dynamic library function cast

1 Answers1