What's the fastest way to list all the exe files in a huge directory in Delphi?

Question

At the moment I'm doing something like this:

var
    Files: TArray<String>;

if IncludeSubDirs then
    Files := TDirectory.GetFiles(Path, '*.exe', TSearchOption.soAllDirectories)
else
    Files := TDirectory.GetFiles(Path, '*.exe', TSearchOption.soTopDirectoryOnly);

Path is a user defined String that can point to any existing directory. For "big" directories with a ton of files and with IncludeSubDirs = True (C:\Windows\ for example) GetFiles takes a very long time (like 30+ secs).

What would be the fastest way to list all the exe files in a "big" directory under Windows with Delphi (if any)?

`GetFiles()` is inefficient at managing the array it returns. You can use `SysUtils.Find(First|Next)()` to avoid that. But, the *fastest* way is to access the underlying filesystem metadata directly, but that is also a lot of manual work. — Remy Lebeau, Feb 02 '22 at 05:07
Keep in mind that the file system itself (as per driver) needs a while to read all that, too. Including the disk's speed when it's not cached yet. While other code is more efficient/faster you still have to expect to wait. — AmigoJack, Feb 02 '22 at 10:54
Whatever process flow this operation is a part of seems flawed. The fastest solution here is to avoid needing to scour the filesystem at all. What problem is this solving? Who ever needs to list all of the .exe files in C:\Windows? And who needs to do it often enough to think that 30 seconds is [too long](https://xkcd.com/1205/)? — J..., Feb 02 '22 at 14:00
@J... It's an edge case. 99% of the time the requested directory contain a handful of files and the snippet above give the result instantly. But if the user input a directory (like C:\ ) by mistake there is no way to abort `GetFiles` midway... I will switch to `FindFirst` and `FindNext` that way you can abort midway... — AlexV, Feb 02 '22 at 14:15
Ah, so the real question is - how to abort a GetFiles operation. You could also just show your users a warning before beginning a search into a known common large directory like the filesystem root or the windows directory, etc. — J..., Feb 02 '22 at 14:15
@J... That and also to get the listing as quickly as possible so that in the ultra rare case that I want to list a huge dir, the result is thrown as quickly as possible :) — AlexV, Feb 02 '22 at 14:17
@AlexV "*there is no way to abort `GetFiles()` midway*" - actually, there is one way. Use one of its overloads that takes a filter predicate, and have it raise an exception when needed. Wrap the call to `GetFiles()` in a `try..except` to catch that exception. — Remy Lebeau, Feb 02 '22 at 15:43

score 0 · Accepted Answer · answered Feb 05 '22 at 17:16

I did some benchmarking and for a huge directory FindFirst / FindNext is about 1.5 to 3% faster than using TDirectory. I would say than both are equivalent in speed (for my use case I saved about 1 sec per minute). I ended up using FindFirst / FindNext since you get results progressively and not all at once, memory management seemed better and it's easier to cancel midway. I also used a TThread to avoid blocking my UI.

This is what I ended up with:

procedure TDirectoryWorkerThread.AddToTarget(const Item: String);
begin
    if (not Self.Parameters.DistinctResults) or (Self.Target.IndexOf(Item) = -1) then
        Self.Target.Add(Item);
end;

procedure TDirectoryWorkerThread.ListFilesDir(Directory: String);

var
    SearchResult: TSearchRec;

begin
    Directory := IncludeTrailingPathDelimiter(Directory);

    if FindFirst(Directory + '*', faAnyFile, SearchResult) = 0 then
    begin
        try
            repeat
                if (SearchResult.Attr and faDirectory) = 0 then
                begin
                    if (Self.Parameters.AllowedExtensions = nil) or (Self.Parameters.AllowedExtensions.IndexOf(ExtractFileExt(SearchResult.Name)) <> -1) then
                        AddToTarget(Directory + SearchResult.Name);
                end
                else if Self.Parameters.IncludeSubDirs and (SearchResult.Name <> '.') and (SearchResult.Name <> '..') then
                    ListFilesDir(Directory + SearchResult.Name);
            until Self.Terminated or (FindNext(SearchResult) <> 0);
        finally
            FindClose(SearchResult);
        end;
    end;
end;

What's the fastest way to list all the exe files in a huge directory in Delphi?

1 Answers1