We have a plugin system that calls functions in dlls (user-generated plugins) by dlopening/LoadLibrarying the dll/so/dylib and then dlsyming/GetProcAddressing the function, and then storing that result in a function pointer.
Unfortunately, due to some bad example code being copy-pasted, some of these dlls in the wild do not have the correct function signature, and do not contain a return statement.
A dll might contain this:
extern "C" void Foo() { stuffWithNoReturn(); } // copy-paste from bad code
or it might contain this:
extern "C" int Foo() { doStuff(); return 1; } // good code
The application that loads the dll relies on the return value, but there are a nontrivial number of dlls out there that don't have the return statement. I am trying to detect this situation, and warn the user about the problem with his plugin.
This naive code should explain what I'm trying to do:
typedef int (*Foo_f)(void);
Foo_f func = (Foo_f)getFromDll(); // does dlsym or GetProcAddress depending on platform
int canary = 0x42424242;
canary = (*func)();
if (canary == 0x42424242)
printf("You idiot, this is the wrong signature!!!\n");
else
real_return_value = canary;
This unfortunately does not work, canary contains a random value after calling a dll that has the known defect. I naively assumed calling a function with no return statement would leave the canary intact, but it doesn't.
My next idea was to write a little bit of inline assembler to call the function, and check the eax register upon return, but Visual Studio 2015 doesn't allow __asm() in x64 code anymore.
I know there is no standards-conform solution to this, as casting the function pointer to the wrong type is of course undefined behavior. But if someone has a solution that works at least on 64bit Windows with Visual C++, or a solution that works with clang on MacOS, I would be most delighted.
