Quicky estimate a number of subfolders

Question

My C# 3.0 application should traverse through folders and do some stuff within. To show a meaningful progress, I need to know total folder count.

If I use Directory.GetDirectories with AllDirectories option, this takes a very long time on my 2Tb hard drive with around 100K folders, and I should present a progress even for that operation! The only meaningful thing I can do is to use recursive Directory.GetDirectories and present a user with a number of already found directories. However, this takes even longer, than the first approach.

I believe, both approaches are too slow. Is there any way to get this number quicker? E.g. take from some file tables using PInvoke? Any other ideas?

score 2 · Answer 1 · answered Jun 20 '11 at 16:13

2

My suggestion would be to simply show the user an infinitely scrolling progress bar while you are getting all of the directories and only when show the user the actual progress while your application does the work.

This way the user will know the application is working in the background while everything happens.

answered Jun 20 '11 at 16:13

Justin Niessner

242,243
40
408
536

Thanks for your comment, Justin. This is easy way to solve the problem. However, as you know, Microsoft guidelines advise to avoid infinite progress bars. It is always good to know how much work is behind. – Alex Jun 20 '11 at 16:14
@Alex - You can only show finite progress when the amount of work is known. When you're unsure, I would rather see an infinite progress bar rather than one that is plain wrong. – Justin Niessner Jun 20 '11 at 16:16
@Alex, consider that Windows shows an infinite progress bar while calculating the number of files for a delete, for example. Guidelines aren't hard and fast rules. I think you can be forgiven for following Windows' (Windows's? Window's?) example in this case. – Greg Jackson Jun 20 '11 at 16:17
Well, the amount of work is pretty good known - take current folder, divide to a total folders and multiply by 100. There is no problem with showing exact progress besides the fact that total folders amount is slow to get. – Alex Jun 20 '11 at 16:18
@Greg - I can agree, but this solution is more "giving up" that an answer to my question. I still hope there is some quick trick. – Alex Jun 20 '11 at 16:19
@Greg Jackson - That's the exact example I was thinking of when I posted this. – Justin Niessner Jun 20 '11 at 16:19
@Alex -- there may be a way to estimate it (I can think of half a dozen possible methods off the top of my head, which generally involve looking at some subpart and estimating the whole from it), but then you risk having something just plain wrong like the Windows 98 progress bars. (see [http://xkcd.com/612/](http://xkcd.com/612/)) – Greg Jackson Jun 20 '11 at 16:23
In general, I think this is a reasonable approach for unknown work, but since it is possible to estimate the work in this specific case, I would rather avoid it. Regarding the wrong progress bars: This should not be an issue since we can just feed in more data as we advance through the files, increasing the accuracy of our estimation in the process. – mafu Oct 12 '12 at 12:57

H H · Answer 2 · 2012-03-04T20:51:47.373

If you implement this you'll find that your first pre-scan was the slowest but it will speed up the next (full) scan because the folder-structure is getting cached.

It may be an option to only count the folders in the first N (2..4) levels. That could still be slow but it will allow for a estimated progress. Just assume all lower levels contain equal numbers of files.

Part 2, concerning the P/Invoke question

Your main cost is here is true lowlevel I/O, the overhead of the (any) API is negligible.

You probably will benefit from replacing GetFiles() with EnumerateFiles() (Fx4). More so for your main loop than for the pre-scan.

Thanks for the comment - it is the same as Gabe already gave and something I also thought about. — Alex, Jun 20 '11 at 16:21

score 1 · Accepted Answer · answered Jun 20 '11 at 16:18

1

This sort of thing is hard to do. If you're just trying to make a rough estimate for a progress bar, you don't need much granularity, right? I would suggest manually traversing the directory tree only one or two levels deep to figure out how many 1st- and 2nd-level subdirectories there are. Then you can update your progress bar whenever you hit one of those subdirs. That ought to give you a meaningful progress bar without taking too much time to compute.

answered Jun 20 '11 at 16:18

Gabe

84,912
12
139
238

Thanks - this is exactly the idea I have. I will choose this approach if I will not find any other option (any magic way to pull out the number quickly from some magic place). – Alex Jun 20 '11 at 16:20
Alex: The filesystem does not store the information you want, so the only way to find it is to scan. – Gabe Jun 20 '11 at 19:19

score 0 · Answer 4 · answered Oct 12 '12 at 12:54

I wrote a pretty simple enumeration of files. The progress is mathematically continuous, i.e. it will not turn to a lower value later on no matter what. The estimation is based on the idea that all folders hold the same number of files and subfolders, which is obviously almost never the case, but it suffices to get a reasonable idea.

There is almost no caching, especially not of deep structures, so this should work almost as quickly as enumerating directly.

public static IEnumerable<Tuple<string, float>> EnumerateFiles (string root)
{
    var files = Directory.GetFiles (root);
    var dirs = Directory.GetDirectories (root);
    var fact = 1f / (float) (dirs.Length + 1); // this makes for a rough estimate

    for (int i = 0; i < files.Length; i++) {
        var file = files[i];
        var f = (float) i / (float) files.Length;
        f *= fact;
        yield return new Tuple<string, float> (file, f);
    }

    for (int i = 0; i < dirs.Length; i++) {
        var dir = dirs[i];
        foreach (var tuple in EnumerateFiles (dir)) {
            var f = tuple.Item2;
            f *= fact;
            f += (i + 1) * fact;
            yield return new Tuple<string, float> (tuple.Item1, f);
        }
    }
}

score 0 · Answer 5 · answered Jun 20 '11 at 16:13

0

Explore FindFirstFile and FindNextFile APIs. I think they will work faster in your case

answered Jun 20 '11 at 16:13

Haris Hasan

29,856
10
92
122

Yeah, but even with that you don't know the number of directories without traversing the whole tree, which is the slow part. – Gabe Jun 20 '11 at 16:15
Good idea - Win API sometimes is really faster than .NET analogue. However, I wonder if such a number is already there in some NTFS/FAT table. – Alex Jun 20 '11 at 16:16
Because they will run natively and most probably Directory class itself would be using something similar. I am not sure though. – Haris Hasan Jun 20 '11 at 16:17
I don't believe this helps much, as .NET is fairly optimized and pretty much accesses the Win32 API directly anyway. Either way there is a context switch, so this probably won't affect the performance much. Also, this does not help with large directory trees. – mafu Oct 12 '12 at 12:50

Quicky estimate a number of subfolders

5 Answers5