40

I've got an application that uses performance counters, that has worked for months. Now, on my dev machine and another developers machine, it has started hanging when I call PerformanceCounterCategory.Exists. As far as I can tell, it hangs indefinitely. It does not matter which category I use as input, and other applications using the API exhibits the same behaviour.

Debugging (using MS Symbol Servers) has shown that it is a call to Microsoft.Win32.RegistryKey that hangs. Further investigation shows that it is this line that hangs:

while (Win32Native.ERROR_MORE_DATA == (r = Win32Native.RegQueryValueEx(hkey, name, null, ref type, blob, ref sizeInput))) {

This is basically a loop that tries to allocate enough memory for the performance counter data. It starts at size = 65000 and does a few iterations. In the 4th call, when size = 520000, Win32Native.RegQueryValueEx hangs.

Furthermore, rather worringly, I found this comment in the reference source for PerformanceCounterLib.GetData:

    // Win32 RegQueryValueEx for perf data could deadlock (for a Mutex) up to 2mins in some 
    // scenarios before they detect it and exit gracefully. In the mean time, ERROR_BUSY,
    // ERROR_NOT_READY etc can be seen by other concurrent calls (which is the reason for the 
    // wait loop and switch case below). We want to wait most certainly more than a 2min window. 
    // The curent wait time of up to 10mins takes care of the known stress deadlock issues. In most
    // cases we wouldn't wait for more than 2mins anyways but in worst cases how much ever time 
    // we wait may not be sufficient if the Win32 code keeps running into this deadlock again
    // and again. A condition very rare but possible in theory. We would get back to the user
    // in this case with InvalidOperationException after the wait time expires.

Has anyone seen this behaviour before ? What can I do to resolve this ?

driis
  • 161,458
  • 45
  • 265
  • 341
  • 2
    Very nice research, +1. Yes, go ahead and worry, the whole perf counter API got unmaintainable afaict. A too simple model that couldn't be bolted on top of the next version of Windows anymore. Vista made a Big Break from it. Good luck with it! – Hans Passant Nov 17 '10 at 22:51
  • Are all Windows versions susceptible to that issue? – Alois Kraus Jun 09 '19 at 18:07

2 Answers2

29

This issue is now fixed, and since there has been no answers here, I will add an answer here in case the question is found in future searches.

I ultimately fixed this error by stopping the print spooler service (as a temporary measure).

It looks like the reading of Performance counters actually needs to enumerate the printers on the system (confirmed by a WinDbg dump of a hanging process, where I can see in the stack trace that winspool is enumerating printers, and is stuck in a network call). This was what was actually failing on the system (and sure enough, opening the "Devices and printers" window also hung). It baffles me that a printer/network issue can actually make the performance counters go down. One would think that there was some sort of fail-safe built in for such a case.

What I am guessing, is that this is cause by a bad printer/driver on the network. I haven't re-enabled printing on the affected systems yet, since we are hunting for the bad printer.

driis
  • 161,458
  • 45
  • 265
  • 341
  • 1
    Is it possible that the reading of performance counters *causes* the print spooler to hang? Firing a second request for some resource that blocks something the first request needs to finish, or somesuch? I suppose this doesn't change the actual cause - a bad driver - but I'd wonder whether the problem won't show up unless you provoke it in the manner you've previously detailed. – Tom W Nov 21 '10 at 12:09
  • 1
    This answer helped us fix a related [new PerformanceCounter()](http://stackoverflow.com/questions/2868068) issue. Removing an old printer prevented the hang and at least showed an error message which we are pursuing. – crokusek Oct 03 '14 at 17:23
  • I'm experiencing a similar issue from an Outlook add-in. Which tool did you use to debug? Currently I have Process Monitor, but I'm having trouble finding the deadlock – public wireless Feb 27 '17 at 17:10
  • @publicwireless WinDbg https://msdn.microsoft.com/en-us/library/windows/hardware/ff551063%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396 – driis Mar 03 '17 at 20:35
  • @driis Thanks. In my case the problem turned out to be Microsoft EMET 5.5 running on the machine – public wireless Mar 08 '17 at 20:47
  • Hi, please. Is there any other way I can manage to make this work? I've already stoped Print Spooler service but with no luck, counter is still hanging. – Fernando Jaconete Dec 09 '21 at 15:20
0

This really didn't help on my case, any operation done that uses the performance Category, will hung in there forever. I am thinking it is more a problem of memory allocation for the call or something related to resources of the machine, I don't have a way to probe it, but trying exactly the same sample call to for example "PerformaceCXounterCategory.Exist" method in a computer with 32GB Ram will run just fine against the one another with only 16GB, if I get a chance to install more memory and test and verify this assumption I will update this ticket

Oscar
  • 697
  • 6
  • 8