High interrupts, sometimes freezes

Question

We have a Win 2003 R2 TS (x86) which has some "bad days". I have not been collecting data on this myself until just now, but it seems most of the time it runs fine, but some days we see high CPU usage, most likely from interrupts.

Using procexp I can see spikes to Interrupts that are not normal compared to other very similar TS'. I don't have a baseline from a "good day" on this particular one, but the problem is there. Interrupts CPU usage range from about 1-10%, mostly at 3-ish %.

Krview shows this (except):

Module           Hits       %Total
intelppm         12195      48%
ntkrnlpa          8994      35%
win32k            1545       6%
hal                984       3%

Normally I see intelppm at >98%, which seem like the normal interrupt-situation.

A zoom on ntkrnlpa shows this:

Module                                Hits   msec  %Total  Events/Sec
NtBuildNumber                          1075      14031     9 %     1915401
RtlCaptureContext                      1062      14031     9 %     1892238
ZwYieldExecution                        991      14031     8 %     1765733
NtFreeVirtualMemory                     803      14031     6 %     1430760
SeMarkLogonSessionForTermination        638      14031     5 %      136768
KeFlushEntireTb                         577      14031     4 %     1028080
KiDispatchInterrupt                     535      14031     4 %      953246
ExAllocatePoolWithTag                   494      14031     4 %      880193
KeAreAllApcsDisabled                    453      14031     3 %      807141
wctomb                                  441      14031     3 %      785760
...

This is where I hit the brick wall... I don't know why these functions are causing interrupts, and I don't know which programs are calling them. Can anyone get me past this point?

We run a dozen accounting applications, some on softgrid (app-v) and MS Office. Taskman doesn't point to any obvious bad processes.

Only time I have seen similar behaviour was with outdated/mismatched vmtools on a VMWare VM. This server runs on a physical blade, the same as about 14 other well behaved TS'.

And no, it's not PIO mode. :-)

Answer 1

Every system call (kernel function) causes an interrupt. I'm not sure these would explain poor performance, though. Also, to see what programs are calling them, it may be best to drill down on particular processes; essentially what you are doing is interrupt analysis on the kernel (ntkrnlpa is the physical address extension NT kernel).

Given that SeMarkLogonSessionForTermination and NtBuildNumber are in such heavy use, it may be that something is causing this server to set up and tear down sessions more often than others, but this doesn't seem to be a likely cause.

What are the symptoms, exactly? It may be a good idea to check the hardware.