2

I need to get the owner for about 100,000,000 windows network files distributed amongst several shares. I'am using advapi32 GetNamedSecurityInfo function from within C# code, but it's a long running process. I'm currently crawling several shares in parallel. And for each share I'm querying every files and folders sequentially. My question, what would be the best approach to minimise the crawl/collection time ?

Thanks for your answers

Cymor
  • 23
  • 2
  • You're already doing things in parallel and have to trawl through 100,000,000 network files. And you have to query each one. If this is too slow for your needs, get someone at your work-place to throw more CPU power on the box you run it from. – Moo-Juice Jul 15 '14 at 17:07
  • @Moo-Juice Usually the limiting factor when crawling file-shares is latency and bandwidth not cpu power :-). – Andreas Jul 15 '14 at 17:43
  • Indeed, cpu will not help. Thanks anyway. – Cymor Jul 17 '14 at 09:32

1 Answers1

0

Ultimately the fastest solution is to query the information locally instead of over the network. Granted, that might not be an option.

File ownership is going to be queried over the network via the SMB protocol. Performance will be very sensitive to latency because lots of requests and responses are going back and forth, and your code is always waiting for a response before sending the next request. Although more recent versions of the SMB protocol support pipelining, your current method of sequential querying per network share means that the pipeline won't be used anyway.

You can probably improve performance by splitting a single network share's requests into multiple threads. I don't know how many concurrent connections are allowed for your particular network shares, but I would test with 8 concurrent requests per server to see how performance is impacted.

RogerN
  • 3,761
  • 11
  • 18
  • Thanks. It looks like the way to go. I don't know when I will be able to try it. Maybe this weekend, I hope so. I will let you know. – Cymor Jul 17 '14 at 09:30