Delphi scanning for database directories using multi threading

Question

I need to scan drives for directories containing my database files to add them to the BDE paths. How can I make my code do it faster using threads? I'm using delphi 2007, so omniThread is not supported. I need to know how to make the thread, and how to execute it. This is my Code: thanks.

procedure TMainFrm.RestoreDBDirs;
var
  Lst: TStringList;
  Dirs: string;
  Counter, j, LstFrom, LstTo: integer;
  SearchRec: TSearchRec;
  ST: TScanThread;    
begin
  Screen.Cursor:= crHourGlass;
  try
    try
      ChangeAlias(AliasCombo);//After this procedure the tables are closed
    except
    end;
    Lst:= TStringList.Create;
    Lst.Clear;
    Counter:= 0;

if Assigned(ChooseDrvFrm) then
with ChooseDrvFrm do
begin
  Lst.Add(lvDrives.Selected.Caption);
  Dirs:= lvDrives.Selected.Caption;

  Progress1.Position:= 0;
  Progress1.Visible:= True;
  stBar1.SimpleText:= 'Searching for Databases...';
end
else
begin
  Lst.Add(GetSystemDrive);
  Dirs:= GetSystemDrive;
end;

repeat
  // Update Progress Bar
  if Assigned(ChooseDrvFrm) then
  with ChooseDrvFrm do
  begin
    Progress1.StepBy(1);
    if Progress1.Position = Progress1.Max then
      Progress1.Position:= 0;
  end;

  Dirs:= Lst.Strings[Counter] +'\';
  if (Dirs <> '.') and (Dirs <> '..')then
    if FindFirst(Dirs +'*.*', faDirectory, SearchRec) = 0 then
    begin
      if ((SearchRec.Attr and faDirectory) > 0)
      and (SearchRec.Name <> '.') and (SearchRec.Name <> '..') then
      begin
        Lst.Add(Dirs + SearchRec.Name);

        if Assigned(ChooseDrvFrm) then
          ChooseDrvFrm.stBar1.SimpleText:= Dirs + SearchRec.Name;
      end;

      while FindNext(SearchRec) = 0 do
        if ((SearchRec.Attr and faDirectory) > 0) and
            (SearchRec.Name <> '.') and (SearchRec.Name <> '..') then
        begin
          Lst.Add(Dirs + SearchRec.Name);
          if Assigned(ChooseDrvFrm) then
            ChooseDrvFrm.stBar1.SimpleText:= Dirs + SearchRec.Name;
        end;
    end;
  Counter:= Counter + 1;
  FindClose(SearchRec);
until Counter = Lst.Count;

Dirs:= '';

if Assigned(ChooseDrvFrm) then
  ChooseDrvFrm.Progress1.Position:= 0;

for Counter:= 0 to Lst.Count - 1 do
begin
  if Assigned(ChooseDrvFrm) then
  with ChooseDrvFrm do
  begin
    Progress1.StepBy(1);
    if Progress1.Position = Progress1.Max then
      Progress1.Position:= 0;
  end;

  if (FileExists(Lst.Strings[Counter] + '\Crt.DB'))
  and (FileExists(Lst.Strings[Counter] + '\Ds.DB'))
  and (FileExists(Lst.Strings[Counter] + '\Turim.DB'))
  and (FileExists(Lst.Strings[Counter] + '\Rprt.DB'))
  and (UpperCase(Lst.Strings[Counter]) <> UpperCase('C:\My Installations\Data'))
  and (UpperCase(Lst.Strings[Counter]) <> UpperCase(ExtractFileDir(ParamStr(0)))) then
  try
    if Assigned(ChooseDrvFrm) then
      ChooseDrvFrm.stBar1.SimpleText:= 'Restoring Databases: '+ Lst.Strings[Counter];

    RestoreAlias(Lst.Strings[Counter]);
  except
    on EDatabaseError do;
  end;
end;

if Assigned(ChooseDrvFrm) then
with ChooseDrvFrm do
begin
  Progress1.Position:= 0;
  Progress1.Visible:= False;
  stBar1.SimpleText:= 'Done';
  MessageDlg('Databases succesfully restored', mtInformation, [mbYes], 0);
  Close;
end;

>     FillAliasCombo;   finally
>     Lst.Free;
>     Screen.Cursor:= crDefault;   end;

More than likely that this is a terrible idea. Not because of the threading, but because of involving huge amount of parsing on nonindexed network resources. Even if you utilise threading you will have to wait untill all your threads finish, and the timeout of accessing a network drive can exceed minutes. — mg30rg, Sep 15 '14 at 12:41
The [front page](http://otl.17slon.com/) says D2007 is supported. — Sertac Akyuz, Sep 15 '14 at 21:32

David Heffernan · Accepted Answer · 2014-09-15T07:06:01.707

3

For traditional spinning disks threading won't help you. Your task is disk bound rather than CPU bound, and threading will lead to inefficient disk head movements. Attempting to do this with multiple threads will likely be slowed than a single thread.

For solid state drives, or network drives then your process is still disk bound. However, using threads to perform tasks in parallel can alleviate some of the latencies of the disk access process and yield improved performance.

It will take some experimentation to work out how best to scan the disks and will likely require different strategies for different disk types.

I think a producer/consumer approach would be the first thing to try. A producer thread that enumerates the directories. And then multiple consumers to read the contents of those directories. You would likely want to batch up multiple directories into a single task to minimize the impact of the threading overhead.

edited Sep 15 '14 at 07:06

answered Sep 14 '14 at 20:03

David Heffernan

601,492
42
1,072
1,490

Thanks. I figured maybe it was possible, because file systems are indexed or something I don't know... Is there perhapse another way? – user1349751 Sep 14 '14 at 20:09
You can try to check this: http://stackoverflow.com/questions/3153428/windows-search-4-query-delphi-example But keep in mind, indexing is platform dependent (before W7 you probably have to use another API) and can be switched off. – Andrei Galatyn Sep 14 '14 at 20:23
Using the index service could be an option. So long as the targets are indexed. – David Heffernan Sep 14 '14 at 21:17
David, Your statement is only true when talking about slower mechanical disks, SSD's on the other hand would benefit from higher IO Queue depths... – whosrdaddy Sep 15 '14 at 06:35
@who Would threading naturally lead to increased queue depths, or would you need to do more? – David Heffernan Sep 15 '14 at 06:48
David, sure why not? Anyway if I where in the OP's shoes, I would do some benchmarking to find out if threading helps or not... (+1 BTW) – whosrdaddy Sep 15 '14 at 08:36
1

It's not necessarily true that multithreading doesn't help in this case. If the directory contents have been accessed before, they will most likely be in the cache so searching won't touch the disk at all. The question that poses itself here is: Why is it too slow in the first place? The way I read that code, it searches the whole drive for directories containing some particular file names. Is that really necessary? Can't you reduce the paths that need to be searched? And regardless of this: How often is that particular routine being used? Does it really make sense to try and optimize it? – dummzeuch Sep 15 '14 at 09:07
Reducing the path seems the safe way to go :) – user1349751 Sep 18 '14 at 09:05

Delphi scanning for database directories using multi threading

1 Answers1