Breaking change in GetFiles with 0.15.2 can't exclude folders

Question

I can't remember where but found an example how to exclude folder from being search. Our issue was search node_modules would cause long path exceptions.

Func<IFileSystemInfo, bool> exclude_node_modules = fileSystemInfo=>!fileSystemInfo.Path.FullPath.Contains("node_modules");

var solutions = GetFiles("./**/*.sln", exclude_node_modules);

Any help to resolve this issue would be helpful.

The example is here: http://cakebuild.net/api/Cake.Common.IO/GlobbingAliases/2CDC8B98 I was really happy to find it as it looked to solve the exact problem I had (node_modules). Perhaps the docs need updating if it doesn't solve this problem? — Justin Caldicott, Jul 24 '17 at 16:04

devlead · Accepted Answer · 2016-08-05T12:48:48.537

To speed up the recursive walking of the file system Cake utilizes the .NET built in features to do so, it's limited by the old 260 char limit of Windows though. So when it's a magnitude faster in most use cases it fails on too deep folder structures like i.e. Node modules can introduce.

You can solve this by iterating folder by folder and applying predicate on which folder to exclude before entering it.

In my example the below folder structure is used

Repo directory
    |   build.cake
    |   test.sln
    |
    \---src
        |   test.sln
        |
        +---proj1
        |   |   test.sln
        |   |
        |   \---node_modules
        |           node.sln
        |
        +---proj2
        |   |   test.sln
        |   |
        |   \---node_modules
        |           node.sln
        |
        +---proj3
        |   |   test.sln
        |   |
        |   \---node_modules
        |           node.sln
        |
        \---proj4
            |   test.sln
            |
            \---node_modules
                    node.sln

What we want is to find all solutions recursively from repo directory not entering the the node_modules directory and not finding the node.sln

Below is suggested solution for this would be to create a utility method called RecursiveGetFile that does this for you:

// find and iterate all solution files
foreach(var filePath in RecursiveGetFile(
    Context,
    "./",
    "*.sln",
    path=>!path.EndsWith("node_modules", StringComparison.OrdinalIgnoreCase)
    ))
{
    Information("{0}", filePath);
}


// Utility method to recursively find files
public static IEnumerable<FilePath> RecursiveGetFile(
    ICakeContext context,
    DirectoryPath directoryPath,
    string filter,
    Func<string, bool> predicate
    )
{
    var directory = context.FileSystem.GetDirectory(context.MakeAbsolute(directoryPath));
    foreach(var file in directory.GetFiles(filter, SearchScope.Current))
    {
        yield return file.Path;
    }
    foreach(var file in directory.GetDirectories("*.*", SearchScope.Current)
        .Where(dir=>predicate(dir.Path.FullPath))
        .SelectMany(childDirectory=>RecursiveGetFile(context, childDirectory.Path, filter, predicate))
        )
    {
        yield return file;
    }
}

The output of this script would be something like

RepoRoot/test.sln
RepoRoot/src/test.sln
RepoRoot/src/proj1/test.sln
RepoRoot/src/proj2/test.sln
RepoRoot/src/proj3/test.sln
RepoRoot/src/proj4/test.sln

This voids the 260 char issue by skipping known trouble makers, won't fix if other unknown paths has same issue.

I'd be happy if default GetFiles behavior was slower but worked in more cases, as above. I expect a number of ppl must be using e.g. NPM, and will have GetFiles calls afterwards. — Justin Caldicott, Jul 24 '17 at 16:09
The Cake.LongPath module will give you support for longer path in Cake. — devlead, Jul 24 '17 at 16:14
Cake.LongPath.Module works nicely for me, thank you! I extended the default build.ps1 to support this: https://gist.github.com/bluebirdtech/6c77bdbab9b61313ed51eb337f2f0b9d — Justin Caldicott, Jul 26 '17 at 06:02

Breaking change in GetFiles with 0.15.2 can't exclude folders

1 Answers1