Windows slowing down with mouse pointer on destop

Question

I've a strange problem. I started approx. 160 processes. Now, if the mouse pointer is on the Desktop, some actions which used to take 100ms, now take 10 seconds although the total load of the system is 13-16%. Even thrid party programs like processhacker slowing down and doesn't refresh their gui. If I move the mouse pointer over some window no matter which one (could be notepad) even the taskbar can help all goes back to normal. Processhacker is refreshing his lists and the responsivness is back to 100ms. Since Microsoft-Support won't help use - since or processes are programmed in Borland-Delphi we have no idea how to find out what's going on here. A colleague tries to reproduce the effect with this little test program:

unit Unit1;

interface

uses
  Forms,
  ExtCtrls,
  Classes,
  Controls,
  StdCtrls;

const
  DEFAULT_INTERVAL = 31;
  MOD_VALUE = 5;
  MOD_INTERVAL = DEFAULT_INTERVAL * MOD_VALUE;
  DEVIATION_BLACK = 2;
  DEVIATION_RED = 10;

type
  TForm1 = class(TForm)
    Label1: TLabel;
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
  private
    Timer: TTimer;
    lastTime: TDateTime;
    procedure OnTimer(Sender: TObject);
    procedure SetLabel(lbl: TLabel);
  end;

var
  Form1: TForm1;
  GCounterT: Integer;

implementation

uses
  Windows,
  Graphics,
  SysUtils,
  DateUtils;

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
begin
  Self.DoubleBuffered := True;

  Timer := TTimer.Create(nil);
  Timer.Interval := DEFAULT_INTERVAL;
  Timer.OnTimer := OnTimer;

  GCounterT := 0;
  lastTime := Now();
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  Timer.Free;
end;

procedure TForm1.OnTimer(Sender: TObject);
begin
  Inc(GCounterT);
  if (GCounterT mod MOD_VALUE) = 0 then begin
    SetLabel(Label1);
    GCounterT := 0;
  end;
end;

procedure TForm1.SetLabel(lbl: TLabel);
var
  newValue: string;
  nowTime: TDateTime;
  msDiff: Integer;
  newColor: TColor;
begin
  if IsIconic(Application.Handle) then Exit;

  nowTime := Now();
  msDiff := MilliSecondsBetween(nowTime, lastTime);
  lastTime := nowTime;

  newValue := Format('TTimer:      %s  dev: %d',[FormatDateTime('ss.zzz', nowTime), msDiff - MOD_INTERVAL]);
  if   (msDiff <= (MOD_INTERVAL + DEVIATION_BLACK))
   and (msDiff >= (MOD_INTERVAL - DEVIATION_BLACK)) then
    newColor := clGreen
  else if (msDiff <= (MOD_INTERVAL + DEVIATION_RED))
   and    (msDiff >= (MOD_INTERVAL - DEVIATION_RED)) then
    newColor := clBlack
  else
    newColor := clRed;
  try
    lbl.Font.Color := newColor;
    lbl.Caption := newValue;
  except
  end;
end;

end.

The effect in not as strong as with the original processes, but it's reproduceable. If one starts 180 of this you can see the same effect only the slowdown is not that severe.

Update Aug 04: I've added a screenshot from a WPA-Analyze-Session. Here one can see the sequence. Starting with mouse on a Window, then Desktop, Window, Desktop and ending with mouse on Window. You can see, that the Thread: CSwitch count is going nearly half if the mouse is on the Desktop. What you also could see is that the system load is between 10-17% the whole time.

You've put six tonnes of load on a single donkey and you're wondering why it can't walk. 31ms timer interval with 160 process = 193 microseconds of CPU time per WM_TIMER message in the queue. If what you're doing in that handler takes more than 193 microseconds, you now have no CPU time left and WM_TIMER messages will become late or dropped entirely. This is not a salvageable design. You have to start again and architect the program differently. — J..., Jul 26 '21 at 15:39
Note the documentation for [WM_TIMER](https://learn.microsoft.com/en-ca/windows/win32/winmsg/wm-timer?redirectedfrom=MSDN) : *The WM_TIMER message is a low-priority message. The GetMessage and PeekMessage functions post this message only when no other higher-priority messages are in the thread's message queue.* That the CPU is struggling to draw (WM_PAINT is higher priority than WM_TIMER) is a dead giveaway that you have no more CPU resources left. 160 processes will saturate all cores on any desktop system... checkmate. You need to rewrite this in a **much** more efficient way. — J..., Jul 26 '21 at 15:47
Also note that every WM_TIMER message here is also causing changes to the UI that require a repaint, so each one will potentially generate a WM_PAINT message also. Since WM_PAINT is higher priority, you have to also deduct the paint time from that 193 microseconds of budget - you are simply way, WAY overloading the computer with too much to do. — J..., Jul 26 '21 at 15:50
And this happens unrelated to which window has the focus and which window is in the foreground? Sadly I don't have _Windows Server 2019_ at hand to reproduce this... — AmigoJack, Jul 26 '21 at 15:51
Which kind of processor(s) do you have in your system? I guess you have 4 sockets equipped with 4 [Intel Xeon Platinum 8368 Processor](https://ark.intel.com/content/www/us/en/ark/products/212455/intel-xeon-platinum-8368-processor-57m-cache-2-40-ghz.html) (For a total of 152 cores or 304 threads). If not, you'd better buy such a system :-) — fpiette, Jul 26 '21 at 15:53
@J... I know I'm nitpicking here, but it's a bit misleading to say the process has only 193 msecs... Single core processor are a relic of the past, especially for servers. Your conclusions are still entirely valid though. — Ken Bourassa, Jul 26 '21 at 15:54
@KenBourassa Yes, fair point, but even on a 28-core server we're still talking about 5ms per WM_TIMER. The code in OP's example takes about 450us on an 10850K hitting 5GHz+ on a single core, plus at least an additional 45us to handle the painting. On a Xeon running flat out at its all-core cap of around half that clock speed, we're at almost 1msec budget per timer message, not even counting everything else the UI thread needs to be doing. I'd guess they're running this on an 8-12 core system, possibly a few years old. We don't know what the actual production code is doing - probably more. — J..., Jul 26 '21 at 16:32
The problem here isn't only the message processing and GUI rendering. The problem here is also the shear fact that OS needs to keep switching between those 180 threads in order to allow all application to do their work. Switching between active threads does bring some additional overhead. That is why when you are making a multi-threaded application you don't just go and create an arbitrary number of threads but only as many threads as you have cores available. — SilverWarior, Jul 26 '21 at 19:32
But just in case if this is a GUI bottleneck you could try to disable AERO Desktop on your windows. Ever since Windows Vista it is possible to bring the Windows GUI to a crawl by simply changing the form caption very fast. — SilverWarior, Jul 26 '21 at 19:36
Thank you all very much for the hints, but no one mention the fact that all this slow down only happens when the mouse is on the desktop. The total load of the real system is only 13-16% and we can reproduce it with server-2016 also. — GreenEyedAndy, Jul 26 '21 at 21:12
@GreenEyedAndy I'm sure an enlightening treatise on the guts of the Windows ecosystem would be interesting as applied to this particular problem, but it doesn't change the solution - hundreds of processes with fast-ticking timers is not a sane design strategy, so you'll need to completely re-architect. Consider first, to what degree those hundreds of processes actually even *need* a UI - surely there are no users watching those labels change colour, so stuff the work into a thread and optionally show a GUI for a specific instance if and only if the user wants to see it. — J..., Jul 26 '21 at 22:07
@GreenEyedAndy I have noticed that fact, but the problem is that your test scenario doesn't do anything that would lead to such behavior. That is why your test scenario doesn't create such strong effect as your actual application. The only ways your test scenario would affect Windows was already discussed in the comments. But in order to help you properly we would need to know what your application is really doing. — SilverWarior, Jul 27 '21 at 07:04
By the way what is the purpose of this text example? Looking at the code it seems as if it is measuring the accuracy of TTimer events being fired. — SilverWarior, Jul 27 '21 at 07:08
@SilverWarior - it's just a demo program to reproduce the effect. Yes it is measuring the accuracy TTimer. My other colleague has found a new effect today. If you right click on the Taskbar and select "Show Desktop" the system also getting fast again. What is windows doing here? I'm sure that J... is absolutly right with his suggestions, but we need to understand this effect BEFORE we do such an expensive redesign. — GreenEyedAndy, Jul 27 '21 at 07:14
@SilverWarior - if we start our processes (181) taskmanager is showing a total of 244 processes and 2780 threads (that doesn't seem like that much) and the effect with the mouse pointer is reproducable. Where is your residence? I'm pretty sure that my CEO is offering a big bounty for finding what's going on ;-) — GreenEyedAndy, Jul 27 '21 at 08:18
*"that doesn't seem like that much"* It wouldn't be "that much" if the processes were non-interactive. Having 181 GUI processes is highly unusual and may cause issues. — Olivier, Jul 27 '21 at 08:30
@Olivier - I fully understand the point with the GUI processes, but what is your explanation for the effect with the mouse pointer over a window? If that's the case the system works as expected. Only if the mouse pointer is on the desktop the system is slowing down. What is windows changing when the mouse pointer is on the desktop instead over any window? — GreenEyedAndy, Jul 27 '21 at 08:40
Honestly I have no idea. Probably MS devs working on Windows would be the best people to ask that question to. — Olivier, Jul 27 '21 at 08:59
@GreenEyedAndy `What is windows changing when the mouse pointer is on the desktop instead over any window?` Windows is changing nothing when mouse cursor is over desktop instead of other window. Now if your applications might be also updating the cursor icon for its window it might force Windows to check if the mouse cursor is over a window that updated its cursor icon to see whether the icon of the actual mouse cursor needs to be updated. I can't think of any other scenario at this time. That is why we need to know what your application is truly doing. — SilverWarior, Jul 27 '21 at 12:09
@GreenEyedAndy The code you have posted does not demonstrate the behaviour you claim is happening. If you can post a real [mcve], we can try to determine what is going on. — J..., Jul 27 '21 at 13:19
@J... - do you have the code compiled with Delphi-Compiler and started the program 180 times? If I do this I can clearly reproduce the effect. The real programs are "talking" to different kind of hardware connected to the network, or over serial ports. How should I make a reproducible example? The minimal example to show the effect is in the question! I could make a video and post it somewhere if you don't thrust me. — GreenEyedAndy, Jul 27 '21 at 14:19
@GreenEyedAndy I can reproduce making the computer slow by launching a large number of these processes, yes. I can also reproduce the system load dropping when the windows are minimized (obviously, Windows does not request anything to repaint when it is not visible, so this makes sense). I cannot reproduce the system load dropping when hovering the mouse over an exposed area of the desktop. At the end of the day, this is junior/student level programming and it will never be suitable for your application. You need a more sophisticated architecture. — J..., Jul 27 '21 at 15:12
@SilverWarior - I've updated the question with a screenshot of a WPA-Analyze. — GreenEyedAndy, Aug 04 '21 at 12:34

GreenEyedAndy · Accepted Answer · 2021-08-16T06:42:11.537

After we managed to add debug-symbols to some of our processes, we found the issue in the Delphi-VCL/Forms.pas. In a new trace, with debug-symbols, we saw that the Application.DoMouseIdle method spends a lot of time finding VCLWindows, get Parents of these and so on. The source of the slowdown is the "FindDragTarget" method. Our processes need no drag'n'drop functionality and they need no hint showing somewhere. So we cut this function call out of the code, which was not easy. Now everything is running fast undependend from the mouse position.

Windows slowing down with mouse pointer on destop

1 Answers1