Application crashes when using hooked function with cout

Question

I'm trying to hook a function using its address, and so far it is overall working great!

The only problem is when using std::cout in combination with having MessageBoxA from the WinAPI hooked, then it crashes! The weird thing is, it only crashes in that specific case, not if it is called in combination with printf or simply int i = MessageBoxA(...);

For testing, I made so that the instructions at the function address directly returns 32. Not much of a hook I know, but this is just for testing.

// mov eax, 32
// ret
const DWORD instructionCount = 6;
BYTE instructions[instructionCount] = { 0xB8, 0x20, 0x00, 0x00, 0x00, 0xC3 };

Besides having to change protection on a region with VirtualProtect(), then now I basically just do

memcpy(MessageBoxA, instructions, instructionCount);

Now testing it using this:

int i = MessageBoxA(NULL, "Hello World", "Injector", MB_OK);

printf("Works: %d\n", i);
printf("Works: %d\n", MessageBoxA(NULL, "Hello World", "Injector", MB_OK));

std::cout << "Works: " << i << std::endl;
std::cout << "Doesn't: " << MessageBoxA(NULL, "Hello World", "Injector", MB_OK) << std::endl;

printf("Hello World\n");

Then it crashes just after std::cout << MessageBoxA(...). Removing that line, and everything works!

Note that it successfully prints 32, it crashes when reaching the next statement.

Again it is only in that case where it doesn't work, so using this:

__declspec(noinline) int testFunction(int i)
{
    return i;
}

Then reusing the above code and changing MessageBoxA to testFunction (as well as the arguments), and now all 4 statements work!

Bottom line, does anybody have any ideas for why and what is causing the crash in that specific case? When the other cases work perfectly fine.

run it in a debugger, and show us the asm, registers, and stack backtrace at the point where it crashes. What calling convention does MessageBoxA use? Is it supposed to pop its args off the stack? If so, the calling code will push args and be expecting the called function to clean up the stack (only applicable to 32bit code, since the win64 ABI puts the first 4 args in regs). — Peter Cordes, Apr 10 '16 at 00:30
Does `std::cout << "Doesn't: " << (int)MessageBoxA(NULL, "Hello World", "Injector", MB_OK) << std::endl;` Work? — Adam Martin, Apr 10 '16 at 00:42

score 3 · Accepted Answer · edited Mar 05 '20 at 10:24

3

I think the issue is that you're corrupting the stack pointer. Try using the following:

const DWORD instructionCount = 8;
BYTE instructions[instructionCount] = { 0xB8, 0x20, 0x00, 0x00, 0x00, 0xC2, 0x10, 0x0 };

That will pop the args off the stack as Peter Cordes mentioned. Hope that helps.

edited Mar 05 '20 at 10:24

vallentin

23,478
6
59
81

answered Apr 10 '16 at 15:15

Aenimated1

1,594
1
9
10

While I understood @PeterCordes comment, I couldn't visualize how to fix it (I never work with assembly). But you answer does the trick! Many thanks! – vallentin Apr 10 '16 at 15:52

Application crashes when using hooked function with cout

1 Answers1