1

I'm writing a low level mouse hook in Windows using the Win32 API, I want to intercept either a button being pressed down or up (still undecided) depending on certain conditions.

To position ourselves let's consider the following snipped of a procedure to handle those hooks:

LRESULT CALLBACK mouseHookProc(int nCode, WPARAM wParam, LPARAM lParam) {
    if (nCode == 0) {
        // Only act on the left click up message:
        if (wParam == WM_LBUTTONUP) {
            // Logic would go here...
            // XXX
        }
    }
    return CallNextHookEx(NULL, nCode, wParam, lParam);
}

It mostly works as expected, when conditions are not met the procedure calls the next in line through CallNextHookEx(), I have a couple where the logic goes too when a disqualifying parameter is found (e.g., if the process that received the button press is not the one I'm looking for).

However, I'm at a loss when it comes to the end of the line, in other words: if I do everything I want to do when a click is registered, how do I signal it? Is there any specific return value the procedure is supposed to return?

For instance, even if programs flow goes through the path where things are handled and everything is dealt with, unless I don't return a call to the next hook the left (or right, depending) button becomes stuck, as if it was never released.

Testing things with the file explorer, if clicking happened over a file you'd be dragging the file around unless the next hook is called. It's a simplistic example of the behavior, Explorer obviously needs to handle the WM_LBUTTONUP itself not to drag the file around, but the question still applies, how does it do it? What's at the end?

XXX in the snippet is where that code would be, everything has been handled so the mouse should be free (unbound to anything?), possibly a return is in order over there; I just can't figure out what.

James Russell
  • 339
  • 1
  • 3
  • 12
  • If you're asking what your function should return if it processes the mouse message, that depends on whether or not you want to suppress the mouse message or not. – Mark Benningfield Aug 13 '21 at 14:15
  • I think so, under those conditions the mouse message would be handled so it should be suppressed. – James Russell Aug 13 '21 at 14:29
  • According to the [docs](https://learn.microsoft.com/en-us/windows/win32/winmsg/lowlevelmouseproc), a function ..."may return a nonzero value to prevent the system from passing the message to the rest of the hook chain or the target window procedure." – Mark Benningfield Aug 13 '21 at 14:33
  • Obviously there is no answer that fit every situation as some application use mouse down, other mouse up, other both. Some capture the mouse or use some internal flags (for ex. `in_drag_mode`). Also timer or drag offset might be used by the other application and there might be other hooks that might too. Thus when you mess with another application expectation, you get what you get like stuck buttons. Why would you mess with File Explorer mouse handling? This is very suspicious. – Phil1970 Aug 13 '21 at 14:42
  • @Phil1970 Nothing malicious :), that was just an example. The main motivation lies with the current state of Windows 11's new contextual menu, because the way to add 3rd party apps to it differs and it's too early for them to think about it I'm missing most of the things that made me productive (e.g. grepWin, archivers, etc.) so I end up clicking on "Show more options" to bring up the old one more often than not. I thought of intercepting the right click to show the old one instead in the meantime, but I'm getting some unexpected mouse behavior (or expected, I'm hooking to be fair). – James Russell Aug 13 '21 at 14:57
  • 1
    You should not mess with the messages in your hook procedure; it is more suitable for getting notified of some events. If you "eat" a message - you will break other apps (as you've noticed). Look into https://learn.microsoft.com/en-us/windows/win32/shell/shell-exts, specifically - https://learn.microsoft.com/en-us/windows/win32/shell/context-menu-handlers – Vlad Feinstein Aug 13 '21 at 16:29
  • When write common UI soft, there are some logic between user code and system, requiring windows message to work, e.g. paint button color darker when pressed down, which is provided by UI framework such as MFC. So if handle the message without notify to other, then the MFC cannot paint button in darker color, which makes the button seems lost response. So when it comes to global message flow, I think is important to figure out where is this handle inserted. Inserting it just before the logic we want to skip and continue to following logic we don't want to skip can be really difficult – wwc Aug 13 '21 at 19:04
  • @wwc You've got things backwards. The Common Controls implementation is a system service. It isn't implemented in a UI framework, like MFC. Instead, MFC provides abstractions over those system services. Message handling is thus largely independent of the library used to implement the UI. – IInspectable Aug 14 '21 at 07:26

1 Answers1

0

I'm going to compile the comments in an answer so those arriving here have a clear view.

@Vlad Feinstein is in the right, one shouldn't mess with the messages in the hook procedure.

With @Phil1970 giving the most accurate answer one could give to the question, because each application handles messages in their own way, there's no general way to know how the handling of one would be; from the outside we know nothing of how they're handled, e.g. which status flags or events are triggered.

In this specific case, I don't think it'd be possible even if one intercepted all of the messages related to mouse clicks, and if it were it'd be way overengineered.

James Russell
  • 339
  • 1
  • 3
  • 12