Performance issues getting list of files (VBA and C# .NET 4.0)

Question

I'm trying to write a piece of code getting a list of files which are present in a folder structure but not present in another folder structure. In other words, I have to archive files only if they are not already archived. The two rootfolders are network folders potentially used by around 1000 users.

In my first attempt, I used VBA from an Excel workbook. I used a shell to get the list and then "Dir" command to check their presence in the other folder.

Here a part of my code:

UPDATE - Here my code:

Dim arr
Dim i As Integer
Dim sFolder As String
Dim sYear As String
Dim sDecade As String
Dim sGFolder As String
Dim sDestXLF As String

sFolder = "T:\FirstRootFolder\"

arr = Filter(Split(CreateObject("wscript.shell").exec("cmd /c Dir ""T:\FirstRootFolder\xx*"" /b /ad /on").StdOut.ReadAll, vbCrLf), ".")

For i = 1 To UBound(arr)
  If Dir(sFolder & arr(i) & "\*.xlf") <> "" Then
    sYear = Right(arr(i), 2)
    sDecade = Mid(arr(i), 3, 2)
    sGFolder = "G:\SecondFolderRoot\" & sYear & "\xx\xx" & sDecade & "\"
    sDestXLF = sGFolder & arr(i) & ".it.xlf"
    If Dir(sDestXLF) = vbNullString Then
      ListBox1.AddItem arr(i)
    End If
  End If
Next i

It works fine and it takes around 6 seconds to complete.

Now, I'm learning framework and C# so I tried to do the same without opening Excel. I tried different solutions but using shell commands (GetFiles, FileExists, ecc) and I end up with this:

string FirstFolderRoot, SecondFolderRoot;
lboLista.DataSource = null;

FirstFolderRoot = @"T:\FirstRootFolder\";
SecondFolderRoot = @"G:\SecondFolderRoot\";

var foundFilesFirst = Directory.EnumerateFiles(FirstFolderRoot, "*.xlf", SearchOption.AllDirectories)
                     .Where(s => s.Contains("xx"))
                     .Select(m => Path.GetFileNameWithoutExtension(Path.GetFileNameWithoutExtension(m))).ToArray();
var foundFilesSecond = Directory.EnumerateFiles(SecondFolderRoot, "*.xlf", SearchOption.AllDirectories)
                     .Select(m => Path.GetFileNameWithoutExtension(Path.GetFileNameWithoutExtension(m))).ToArray();

var foundFiles = foundFilesFirst.Except(foundFilesSecond);
lboLista.DataSource = foundFiles.ToList();

It works fine too but it take around 1 minute and half to complete, with most of the time spent filling the two collections.

Is there a way to have comparable performances to VBA? Are really shell commands so faster than framework or it's me not using it in the right way?

I read that the faster way would be using winapi but I really wish to use framework.

Answer 1

Take a look at the example here: msdn link for an example of directory and file comparisons in C#.NET. It makes use of IEqualityComparer() and SequenceEqual()

The bottleneck will be the time taken to run the two GetFiles() commands. If these are different network drives these can be run in separate threads for increased performance.

For clarity here's a slightly modified excerpt:

// This implementation defines a very simple comparison 
// between two FileInfo objects. It only compares the name 
// of the files being compared and their length in bytes. 
class FileCompare : System.Collections.Generic.IEqualityComparer<System.IO.FileInfo>
{
    public FileCompare() { }

    public bool Equals(System.IO.FileInfo f1, System.IO.FileInfo f2)
    {
        return (f1.Name == f2.Name &&
                f1.Length == f2.Length);
    }

    // Return a hash that reflects the comparison criteria. According to the  
    // rules for IEqualityComparer<T>, if Equals is true, then the hash codes must 
    // also be equal. Because equality as defined here is a simple value equality, not 
    // reference identity, it is possible that two or more objects will produce the same 
    // hash code. 
    public int GetHashCode(System.IO.FileInfo fi)
    {
        string s = String.Format("{0}{1}", fi.Name, fi.Length);
        return s.GetHashCode();
    }
}


class CompareDirs
{

    static void Main(string[] args)
    {
        // Create two identical or different temporary folders  
        // on a local drive and change these file paths. 
        string pathA = @"C:\TestDir";
        string pathB = @"C:\TestDir2";

        System.IO.DirectoryInfo dir1 = new System.IO.DirectoryInfo(pathA);
        System.IO.DirectoryInfo dir2 = new System.IO.DirectoryInfo(pathB);

        // Take a snapshot of the file system.
        IEnumerable<System.IO.FileInfo> list1 = dir1.GetFiles("*.*", System.IO.SearchOption.AllDirectories);
        IEnumerable<System.IO.FileInfo> list2 = dir2.GetFiles("*.*", System.IO.SearchOption.AllDirectories);

        //A custom file comparer defined below
        //Use Except to retrieve files from list2 not in list1
        var uniqueFiles = list1.Except(list2, new FileCompare());
   }

}