0

I'm using a local hook (WH_KEYBOARD) with ms word (OpusApp). Well, as far as I know a 32bit app with a 32bit DLL must work only with 32bit target applications. The weird thing is that the program only works with 64bits apps!!! That is it, only with 64bits APPS! For example, it works with IE 64 but not with IE 32! The app and dll are 32bit compiled with radstudio XE2, I confirmed the version into PE header. In 32bit OSs, the app and dll doesn´t work.

I found no solutions on net and see no starting point to solve this weird problem.

The DLL code:

// Exported functions

extern "C" __declspec(dllexport)bool __stdcall InstallMouseHook(unsigned long, void *);

extern "C" __declspec(dllexport)bool __stdcall RemoveMouseHook();

// Callback Procedure Declaration

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam);

// Global variables

HHOOK HookHandle;
HINSTANCE DllInstance;
typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM);
CALLIT callIt = NULL;

int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
{
  DllInstance=hinst;
  return 1;
}

bool __stdcall InstallMouseHook(unsigned long pid, void *function)
{

     callIt = ( CALLIT ) function;

     if (function == NULL) {

        ShowMessage("function is null!");

     } else if (callIt == NULL) {

        ShowMessage("callIt is null!");

     }  

     HookHandle=SetWindowsHookEx(WH_KEYBOARD ,reinterpret_cast<HOOKPROC> (HookProc),DllInstance,pid);

    if (HookHandle==NULL)return false;

    else return true;

}

bool __stdcall  RemoveMouseHook()
{
  if(UnhookWindowsHookEx(HookHandle)==0)
  {
    return false;
  }
  else return true;
}

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam)
{
    if (code<0) {
        return CallNextHookEx(HookHandle,code,wParam,lParam);
    }

    if (callIt != NULL) {
        callIt(code,wParam,lParam);
    } else {
        ShowMessage("HookProc - no function to execute OR 32/64 bits problem!");
    }

  //Call the next hook in the chain
  return CallNextHookEx(HookHandle,code,wParam,lParam);
}

The EXE calling code:

void __fastcall TfrmMouseHook::btnHookAppDllClick(TObject *Sender)
{
    HWND hWindow;
    unsigned long pid;

    String s = "MouseHookDLL.dll";
    DllHandle=LoadLibrary(s.w_str());
    MOUSEHOOKFCT_2 InstHook=reinterpret_cast<MOUSEHOOKFCT_2> (GetProcAddress(DllHandle,"InstallMouseHook"));

    hWindow = FindWindow(ComboBox1->Text.w_str(),NULL);

    if (!hWindow) {
        msg("hWindow fail");
        return;
    }

    pid = GetWindowThreadProcessId(hWindow ,0);
    if (!pid) {
       msg("pid fail");
       return;
    }

    if(!InstHook(pid, (void *) callIt )) {
        msg("Unable to install  hook!");
    } else {
        msg(" #### hook INSTALLED! ####");
    }


}

CALLIT callIt(code,wParam,lParam) {
    frmMouseHook->msg("hook callit: code="+IntToStr(code) +" wparam="+IntToStr(wParam)+" lparam="+IntToStr(lParam) );
}



   Call IT is a function pointer to a hooker app function.

    Any ideas will be very wellcome!
Johan
  • 74,508
  • 24
  • 191
  • 319
sgm
  • 196
  • 3
  • 14
  • 1
    Clearly the operating system is offended that you wrote a MouseProc() for a keyboard hook and decided to invert everything. – Hans Passant Sep 03 '12 at 19:14
  • Well, I'm a bizarre person too! I will try apologize OS, but I think it will not solve the problem. – sgm Sep 03 '12 at 20:42
  • You need to show more code. Where is your call to SetWindowsHookEx()? Is it inside the app or inside the DLL? What parameters are you passing to it? – Remy Lebeau Sep 20 '12 at 22:51
  • 1
    Your code has some bugs in it. You need to store the `HHOOK` handle in a block of shared memory. You can use `CreateFileMapping()` and `MapViewOfFile()` for that. But more importantly, your `callIt()` callback in the app is not using the `__stdcall` calling convention but the DLL is expecting it to. You would have caught that error if your code were type-safe. The `function` parameter of `InstallMouseHook()` needs to declared as `CALLIT` instead of `void*`. – Remy Lebeau Sep 23 '12 at 19:52
  • 1
    Another problem is that you are installing a callback function pointer that belongs to one process but might be executed by the process being hooked. A better solution is to use an `HWND` instead of a function pointer and have the hook send a message to the window instead of calling the function pointer directly. That would be much safer. – Remy Lebeau Sep 23 '12 at 19:57

2 Answers2

1

It is physically impossible for a 32-bit app to install a 32-bit hook DLL and have it executed in 64-bit processes. A 32-bit DLL simply cannot be injected into a 64-bit process. Period. MSDN says this in multiple places, including in the SetWindowsHookEx() documentation:

SetWindowsHookEx can be used to inject a DLL into another process. A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL cannot be injected into a 32-bit process. If an application requires the use of hooks in other processes, it is required that a 32-bit application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit processes, and a 64-bit application call SetWindowsHookEx to inject a 64-bit DLL into 64-bit processes. The 32-bit and 64-bit DLLs must have different names.

Because hooks run in the context of an application, they must match the "bitness" of the application. If a 32-bit application installs a global hook on 64-bit Windows, the 32-bit hook is injected into each 32-bit process (the usual security boundaries apply). In a 64-bit process, the threads are still marked as "hooked." However, because a 32-bit application must run the hook code, the system executes the hook in the hooking app's context; specifically, on the thread that called SetWindowsHookEx. This means that the hooking application must continue to pump messages or it might block the normal functioning of the 64-bit processes.

If a 64-bit application installs a global hook on 64-bit Windows, the 64-bit hook is injected into each 64-bit process, while all 32-bit processes use a callback to the hooking application.

The fact that you say your app and DLL do not work on 32-bit OS versions suggests your hooking code is flawed to begin with. But you have not shown enough code to diagnose that one way or the other.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • What could I say, the above code (compiled for 32 bits with rad studio) are working only with IE64, not with IE32 (same with others apps). The function SetWindowsHookEx is inside the DLL. – sgm Sep 22 '12 at 23:43
  • There is absolutely no possible way that IE64 can be executing your 32-bit hook DLL. It is physically impossible. Only 32-bit processes can invoke 32-bit hooks. Period. – Remy Lebeau Sep 23 '12 at 19:58
  • Thanks Remy for the answer. callIt uses __stdcall in typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM); I understand that is not possible to a 32bit process, with a 32bit dll hook a 64bit application. But, unless radstudio are building with 64bit (in my version there are only option to build for 32bit windows) or Internet Explorer (64-bit) shortcut are point to the 32bit version, this bizarre thing are hapenning. See desktop images [link](http://www.smiranda.hostoi.com/hook.jpg) In fact, call a function pointer in another process is bizarre ... but this works as a first try... – sgm Sep 24 '12 at 21:16
  • In the EXE code, you declared the `callIt()` callback as `CALLIT callIt(code,wParam,lParam)`, which is completely wrong. It needs to be declared as `void __stdcall callIt(int code, WPARAM wParam, LPARAM lParam)` instead. Again, your compiler would have caught that had you written type-safe code to begin with. – Remy Lebeau Sep 24 '12 at 21:38
  • C++Builder does not support development of 64-bit executables yet. I guarantee you that 64-bit IE is NOT invoking your hook. It can't. It is a 64-bit process, and a 64-bit process CANNOT invoke a 32-bit hook. Make ABSOLUTELY sure that you are actually "hooking" IE 64. For instance, when I run IE on my 64-bit Win7 machine, it actually runs 32-bit versions of IE instead. – Remy Lebeau Sep 24 '12 at 21:40
  • By default, Microsoft usually runs the 32-bit version of IE on a 64-bit machine, depending on how you are launching it. This is [by design](http://support.microsoft.com/kb/896457). – Remy Lebeau Sep 24 '12 at 21:48
  • Well, the problem is deeper. Besides I have cited only IE, the program works fine with C:\Program Files\Microsoft Office\Office14\WINWORD.EXE too. I tested the PE header, it is AMD64. In fact, I spent two days until realize why the program are working in my note (with office 64) and not in my desk (with office 32) {against logic}. Till now, I could say the program works fines with some apps and dont works at all in 32bit windows OS. I have build an App to search PE header too. Program and DLL link: [link](http://www.smiranda.hostoi.com/HookProgram.zip) – sgm Sep 26 '12 at 02:49
0

What happens? Besides MSDN or anyone else says, there are some bug in XE6, compiling DLL in newer versions of IDE make this behavior disappears, in fact, the new DLL crash and hook nothing.

As Remy noted, by test I passed a function pointer to the DLL, one wrong thing to do but, when added with the wrong thing done by Embarcadero, sort of functioned.

By now, and I know people will get mad, I put both methods (the wrong and the correct hooks) in the same DLL and in my application and... get crazy... could hook into 32 and 64 bits app with only one DLL.

Don't believe? Install XE6 and try!

And Works in windows 10 as well in windows 7.

sgm
  • 196
  • 3
  • 14