Extreme slowdown when changing folder permissions

Question

Last Friday I attempted to set permissions on a folder containing multiple folders. I simply deleted one user group's access entry, then clicked Apply and was greeted with an hourglass. Pretty soon I couldn't do anything on the server without extremely lag, and started getting phone calls from users. It lasted a few minutes until I manually hit the power button, because I couldn't even log in anymore, even sitting right at the console--it soft-froze on 'applying desktop settings'.

In the sys log, we have hundreds of these entries per second starting right when I hit Apply and ending with my manual reset:

Event Type:    Error
Event Source:    Srv
Event Category:    None
Event ID:    2000
Date:        2010-11-26
Time:        8:55:01 AM
User:        N/A
Computer:    MyServer
Description:
The server's call to a system service failed unexpectedly.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 01 00 54 00   ......T.
0008: 00 00 00 00 d0 07 00 c0   ....Ð..À
0010: 00 00 00 00 0a 01 00 c0   .......À
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 34 03 bd 00               4.½.

But only certain seconds. Seemed to have gone in 1-4 second bursts, but still most seconds in a given minute are hit with at least a dozen entries.

Now, I still need to apply this change, but all I could find on Google did not seem to be solutions, just other people saying this or that permissions change took 1 or 2 or 22 or 28 hours of such extreme slowness. I need to make this change at some point soon and I'm wondering what I can do to increase my chances that it won't have to wait until the weekend or Christmas or I don't know what. Our server is not the newest but not too shabby (Dual Intel Xeon 3.2 GHz, 4 GB RAM, around a dozen clients) and is otherwise snappy, especially after recently kicking NetBIOS and other Win98 legacy features off our network.

Does the above event log entry indicate something I could fix to speed the whole process up? (And avoid filling my event log up!)

Update w/kern To answer tony roth's question, it says:

PathName                                  ServiceType    Started  
C:\WINDOWS\system32\DRIVERS\ACPI.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\afd.sys       Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\atapi.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ati2mpad.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\audstub.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\Beep.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\cdrom.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\compbatt.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\cpqasm2.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\cpqcidrv.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\CPQCISSE.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\cpqcissm.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\crcdisk.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\disk.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\dmio.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\dmload.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\Fips.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ftdisk.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\msgpc.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\hidusb.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\i8042prt.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\imapi.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\intelide.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\intelppm.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ipsec.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\isapnp.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\kbdclass.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\KSecDD.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\mnmdd.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\mouclass.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\MountMgr.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\mssmbios.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\NDIS.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ndistapi.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ndiswan.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\NDProxy.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\netbt.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\npf.sys       Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\Null.sys      Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\PartMgr.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\pci.sys       Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\PCIIde.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\raspptp.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\ptilink.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\b57xp32.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\rasacd.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\rasl2tp.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\raspppoe.sys  Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\raspti.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\RDPCDD.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\rdpdr.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\RDPWD.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\redbook.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\serenum.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\serial.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\swenum.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\sysmgmt.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\tcpip.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\TDTCP.sys     Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\termdd.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\update.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\usbehci.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\usbhub.sys    Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\USBSTOR.SYS   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\usbuhci.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\drivers\vga.sys       Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\volsnap.sys   Kernel Driver  TRUE     
C:\WINDOWS\system32\DRIVERS\wanarp.sys    Kernel Driver  TRUE

If you right-click on the folder, how many files/folders are in the folder? — Greg Askew, Nov 30 '10 at 00:53
My RDBMS on the same machine can delete that many rows without bringing the server to its knees, and also estimate how long it might be (under 30 seconds.) But in any case, what's causing the event log errors? — Kev, Nov 30 '10 at 01:16
Comparing RDBMS operations to file system operations is comparing apples to automobiles. — joeqwerty, Nov 30 '10 at 02:58
Shouldn't be. Some FSes are built upon RDBMSes. In any case, I don't think such a standard operation should make the server unusable, especially since the CPU and memory usage don't even budge, let alone spike, while this is happening. And if it must, it should warn you, and also provide a time estimate. In any case...whether it's thought that I ought to expect more from the technology or not...how is one to solve this problem? — Kev, Nov 30 '10 at 03:19
Some FS'es may be built on RDBMS but NTFS is not, so again you're comparing apples to automobiles. You're changing the ACE's on the ACL's of more than half a million file system objects. I would expect that to take a while. It's hard to give you any type of estimate but based on similar operations that I've performed I would guess that it's going to take more than 4 minutes but less than 4 hours. Also, you're probably not going to see any noticable dings to overall CPU and memory usage but I would expect to see some high disk metrics as well as seeing the file system cache getting hammered. — joeqwerty, Nov 30 '10 at 03:35
one thing a rdbms system does not have to contend with is 3rd party filter drivers as in AV software which temporarily disabling is my #1 troubleshooting step on file server issues. — tony roth, Nov 30 '10 at 03:49
@joeqwerty, thanks for your estimate. @tony roth, it was over terminal services. Does it make a difference? Good point about disabling AV; however, this particular server has only scheduled scans. — Kev, Nov 30 '10 at 15:56
tscon is fine and you were working with the drive directly as in f: not \\servername\sharename? — tony roth, Nov 30 '10 at 16:09
I went to do this change again last night, and found that all of the permissions I would have applied were already applied, even though earlier (after the restart) I had found that some were and some weren't. I'm curious as to when it decided to apply them, but I'd also still like to figure out what caused all these errors. — Kev, Dec 01 '10 at 16:48

score 3 · Accepted Answer · answered Nov 30 '10 at 04:48

3

The comments on your question have it right, basically.

You're touching the ACEs on all the subfolders and files below the point you're changing an ACL because the API the shell calls does it.

The error you're getting in your event log from SRV.SYS is an "STATUS_PROCESS_IS_TERMINATING" (0xc000010a). It's unclear to me why you're seeing that particular error. Given that it coincided with your change to the ACL it's probably related but isn't something I've ever seen on ACL changes to deep file hierarchies on live file servers, personally.

I've changed ACLs on deeper hierarchies on live file servers w/ slower specs and vastly more active w/o issue. I suspect, as Tony Roth comments, that some third-party software (like anti-virus) may be at play in the behavior you're seeing.

answered Nov 30 '10 at 04:48

Evan Anderson

141,881
20
196
331

How did you determine it's 0xc000010a? I only see the pattern 0xc0000000000a01 in there. – Kev Nov 30 '10 at 15:59
The only "deep" software that might have been running (minimized to tray, though) would be SysInternals' Process Explorer. I'll try again without that. – Kev Nov 30 '10 at 16:01
you may be only scanning on a scheduled basis but the AV filter is doing work all the time! – tony roth Nov 30 '10 at 16:11
It doesn't have a filter. It's ClamAV for Win (the old one, not the new cloud-based ClamWin.) – Kev Nov 30 '10 at 16:14
1

@Kev: You know, I'm not sure how I know that the DWORD at offset 0x14 in the data returned in an SRV event log entry is the NTSTATUS code for the error, but I do. Your event log entry is formatted as bytes, but the little-endian DWORD value at offset 0x14 is "0xc000010a". (I may know this from reading, like, "Windows NT Internals Revealed" or some such... I don't recall where I learned it.) You can see Microsoft referencing the status code in an SRV event log entry here: http://support.microsoft.com/kb/136150 – Evan Anderson Dec 01 '10 at 20:03
@Evan Anderson, +1 thanks for the info and link! – Kev Dec 01 '10 at 22:49

tony roth · Answer 2 · 2010-12-01T17:04:04.853

1

whats the result of running the following

wmic sysdriver where "servicetype like 'kern%' and started ='true'" get pathname,started,servicetype

edited Dec 01 '10 at 17:04

answered Nov 30 '10 at 16:59

tony roth

3,884
18
14

dang wrong query see edited answer – tony roth Dec 01 '10 at 17:03
No worries, updated again. – Kev Dec 01 '10 at 22:52
@Kev: I don't see any drivers there that don't look familiar... Hrmph... – Evan Anderson Dec 02 '10 at 01:07
a rootkit would hide everything so I'm starting to wonder about this particular server. Clamav is not really sufficient in anyway and diffently not when its a scheduled scan. Also anything thats demand start won't show up in my wmic query so you could remove the started='true' but I starting to think this path is a waste of time. Kinda confused about this one! – tony roth Dec 02 '10 at 02:49
Well, neither was Symantec Corporate. Their updater broke at one point and we ended up with thousands of viruses. Plus it slowed everything down noticeably. There are sysadmins out there who argue that you can do without an antivirus package, even on Windows. – Kev Dec 02 '10 at 17:57
I just ran the OneCare safety scanner and it gave me a clean bill of health, if that means anything. – Kev Dec 03 '10 at 14:20
a true rootkit is undiscoverable within the os! Its true you could run without AV software but you'd have to be unplugged from the network. Its also true that I ran xp without av software for 10 years the only protection I had was a natted router. The trick is to never elevate inappropriately... – tony roth Dec 07 '10 at 17:47

Extreme slowdown when changing folder permissions

2 Answers2