5

I've built a DLL that gets injected into a console application usually via SetWindowHookEx. Its important for the DLL to output information to the console which I have been doing with std::cout. The DLL was nearing completion until I tried building the DLL in release mode which rendered all cout lines useless. I've verified the the DLL is injecting and is executing by doing a simple null dereference that causes the program to crash in the dllmain function. Same story with std::printf.

void onAttach()
{
    //WARNING THIS IS A DEMONSTRATION
    std::cout<<"test"<<std::endl;
    //int* intPtr = 0;
    //*intPtr = 3; //This causes a crash
}

// entry point
BOOL WINAPI DllMain ( HMODULE hModule, DWORD dwReason, LPVOID lpReserved )
{
    switch ( dwReason )
    {
        case DLL_PROCESS_ATTACH:
            DisableThreadLibraryCalls ( hModule );
            CreateThread ( NULL, 0, ( LPTHREAD_START_ROUTINE ) onAttach, NULL, 0, NULL );
            return true;
        break;

        case DLL_PROCESS_DETACH:
            return true;
        break;
    }
}

I really don't know how to approach this problem. Is the Release linker somehow excluding dependencies?

I'm using MSVS 2010 and default release/debug configuration setup. The debug dll is about 5,137kb and the release dll is only 23kb.

Slight
  • 1,541
  • 3
  • 19
  • 38
  • 4
    I suspect `stdout` is being flushed in debug but not in release. Try `std::cout << "Test" << std::endl;` in release. – johnsyweb Jan 08 '13 at 05:15
  • Same story with cout using endl unfortunately. – Slight Jan 08 '13 at 05:21
  • 2
    The `printf` call doesn't flush the output. It also uses different buffers from `std::cout`, so you really shouldn't mix these. – Some programmer dude Jan 08 '13 at 06:13
  • 2
    Are you linking against the DLL version of the C runtime ([`/MD`](http://msdn.microsoft.com/en-us/library/2kzt1wy3%28v=vs.71%29.aspx))? Can you print to the console if you use `WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), "test", 4, NULL, NULL)` instead of using `printf` or `cout`? – Adam Rosenfield Jan 08 '13 at 06:40
  • @AdamRosenfield Ah, thank you. WriteFile works. Any theory as to why this is? I've tried calling std::cout.flush() though the newline should do it. – Slight Jan 08 '13 at 10:21
  • Newline does not cause a flush! – johnsyweb Jan 08 '13 at 23:53

2 Answers2

0

You should not do anything in DllMain which depends on another DLL. You violate that at least three times: printf and std::cout depend on the CRT (which in turn depends on Win32 DLLs) and CreateThread depends on Kernel32.DLL.

DllMain is intended for simple things like zeroing variables etc.

MSalters
  • 173,980
  • 10
  • 155
  • 350
  • 2
    *sigh* This is just some basic code to get the point across. Please use your imagination and pretend like its inside the onAttach function would you? Not to mention this does not answer the question in the least. I appreciate all answers but this seems more like a jab than an answer. – Slight Jan 08 '13 at 10:12
  • 1
    @Slight: The behavior of code in DllMain is very unpredictable and critically dependent on DLL load order. Since Debug builds will depend on different DLLs (e.g. ToolHelp.DLL), it's almost a given that the DLL load order differs. Therefore, just having such code in DllMain is a sufficient explanation for the observed behavior. And I obviously can't provide answers based on code you didn't show us. – MSalters Jan 08 '13 at 12:02
  • 3
    @salters My code is being injected. Henceforth you can assume my DLL is loaded last. I also mentioned that I had std::cout usage throughout my project. Your post is more "advice" then an answer which you admitted in your last sentence. – Slight Jan 09 '13 at 02:51
0

I know @salters is correct in stating that no calls to other libraries should be made in a DLL, but after trial and error I've found that compiling the DLL's in x64 configuration (instead of x86) the DLL's can "cout" or show use the "MessageBox" function without any issues (can likely call functions from a number of other libraries). Hope this helps :).

Ice Phoenix
  • 1,001
  • 3
  • 16
  • 34