1

I need to pump COM messages while waiting for an event to fix a deadlock. It's better to pump as few messages as possible just to process that COM call. The best candidate for this role is CoWaitForMultipleHandles but starting from Vista it pumps WM_PAINT in addition to COM messages. Pumping WM_PAINT is too dangerous for me from re-entrance perspective and I don't want to install a custom shim database as a solution for this problem.

I'm trying to pump COM messages sent to the hidden message-only window manually.

I have found two ways to get HWND of the hidden window:

  1. ((SOleTlsData *) NtCurrentTeb()->ReservedForOle)->hwndSTA using ntinfo.h from .NET Core. This seems to be undocumented and not reliable solution in terms of future changes.
  2. Find window of OleMainThreadWndClass as suggested in this question. The problem is that CoInitialize does not create the window. It is created later on first cross-apartment call which may or may not happen in my application. Running the search loop every time I need HWND is bad from performance perspective but caching HWND seems impossible because I don't know when it's created.

Is there a way to determine if the hidden window is created for the current apartment? I suppose it will be cheaper than the loop and then I could find and cache HWND.

Is there a better way to pump COM messages without pumping WM_PAINT?

Update: you can force the window creation by calling CoMarshalInterThreadInterfaceInStream for any interface. Then call Co­Release­Marshal­Data to release the stream pointer. This is what I end up doing along with the search for OleMainThreadWndClass.

Dmitry Oksenchuk
  • 121
  • 1
  • 4
  • 1
    If you want to go for dark routes, you should be able to tweak AppCompatFlags directly through in memory PEB : https://social.msdn.microsoft.com/Forums/sqlserver/en-US/5a28a9f5-5711-4efa-843e-e98927fa2b92/dcom-in-vista-specifically-processing-wmpaint-messages?forum=windowsgeneraldevelopmentissues .I think you can disable WM_PAINT pump if you binary-and current AppCompatFlags with flag value 0x100000. Note AppCompatFlags offset from PEB beginning is different from x64 and x32. – Simon Mourier Oct 18 '17 at 07:30
  • I can't think of a single good reason to not process WM_PAINT messages. If something is "dangerous" in your WM_PAINT handler, I would guess that you're doing something in WM_PAINT that you shouldn't be doing. WM_PAINT handler should only ever read state (any variables necessary to figure out what to draw), not modify state. – Michael Gunter Nov 03 '17 at 15:53
  • @MichaelGunter, virtual grids do a lot of interesting things in WM_PAINT handlers. – Dmitry Oksenchuk Nov 07 '17 at 13:47

1 Answers1

-3

WM_PAINT is generated when there is no other message in the message queue and you execute GetMessage or use PeekMessage.

But WM_PAINT is only sent if you Dispatch it. Also there is no new WM_PAINT message until a window is invalidated again.

So it depends on you if you dispatch a WM_PAINT message or not. But be aware, there are other chances of reentrances like a WM_TIMER message.

The details about this are in the docs for WM_PAINT.

From my point of view the best solution would be to set you application in a "wait" mode, that even can handle WM_PAINT in this undefined waiting state. You know when you are reentered. It is always after a WM_PAINT... or similar messages that arrive like other input messages. So I don't see any problems here. An STA has one thread and you always process messages to an end, until you execute GetMessage, launch a modal dialog or show a MessageBox. When you are inside some message handling, nothing will disturb you.

Maybe an other solution would be to wait inside a second thread for this event. This thread may not have any windows and you can translate the event to anything you need in your application.

So you question may not have enough information how this deadlock really appears. So this answer may not be sufficient.

After writing als this I tend to the opinion that this is an XY problem.

xMRi
  • 14,982
  • 3
  • 26
  • 59