How can i get all files from disk?

Question

I have to create a vector of all files on my disk, starting from root ("C:/"). I used filesystem library and recursive_directory_iterator, however program throws filesystem::filesystem_error when it reaches some of system directories which only the system has access to.

#include <filesystem>
#include <iostream>
#include <vector>

using namespace std;
namespace fs = std::filesystem;

vector<fs::path> get_all_files_from_disk(fs::path path,vector<fs::path>* all_files = NULL) {
    if (all_files == NULL)
        all_files = new vector<fs::path>();
    try {
        for (auto dir : fs::directory_iterator(path)) {         
                if (dir.is_directory())
                    *all_files = get_all_files_from_disk(dir, all_files);
                else
                    all_files->push_back(dir.path());
        }
    }
    catch (fs::filesystem_error e) {
        cout << "error";
    }
    return *all_files;
}

vector<fs::path> getAllFilesFromDisk(fs::path path) {
    vector<fs::path> all_files;
    for (auto dir : fs::recursive_directory_iterator(path)) {
        if (!dir.is_directory())
            all_files.push_back(dir.path());
    };
    return all_files;
};

int main() {
    setlocale(LC_ALL, "");
    fs::path path("C:/");
    vector<fs::path> a = get_all_files_from_disk(path);
    vector<fs::path> b = getAllFilesFromDisk(path);
}

The first algorithm doesn't crash, but executes really long time (it executed at about 40 minutes), however when output displaying all elements of vector is empty for some reason, with no-root directories it works fine. The second one crashes because of the reason stated in the beginning.

:( Could you help please?

Unrelated, but why `new`? This is not Java, you have a memory leak now. Just return by value and `move` into functions. Also, did you try debugging the code? — Quimby, May 08 '22 at 13:16
If you don't have permission to access certain directories/files, then there is nothing you can do about it. Ignore the exceptions and continue. That's the best you can do. What you are doing with `all_files` looks wrong. Just create one vector as you do in the second function and then append the results of the recursive calls to it. — user17732522, May 08 '22 at 13:33
@dummygum `*all_files = ` overwrites the vector each time. That doesn't seem to be what you want. Then you also leak its memory at the end of the first recursion. Just use a normal automatic-storage duration vector in each recursion and use `.insert` to insert the elements from the recursive call. That's technically not as efficient as using a single vector, but it won't matter relative to the time IO takes. — user17732522, May 08 '22 at 13:47

score 0 · Answer 1 · answered May 08 '22 at 15:34

There may be the problem that the underlying filesystem denies you the permission to read certain directories (like for example special system directories).

That my be the reason that you iterator fails.

If you read about the std::filesystem::recursive_directory_iterator in the CPP reference here, then you can see in the description of the constructor no 5, that you can set a flag to ignore unaccessable directories. Please read here about the flag skip_permission_denied

Still it will run some time, because of many many many directories is you start from root. Please Switch optimizations on.

Example code:

#include <filesystem>
#include <iostream>
#include <vector>

using namespace std;
namespace fs = std::filesystem;

vector<fs::path> getAllFilesFromDisk(fs::path path) {
    vector<fs::path> all_files;
    for (auto dir : fs::recursive_directory_iterator(path, fs::directory_options::skip_permission_denied)) {
        if (!dir.is_directory())
            all_files.push_back(dir.path());
    };
    return all_files;
};

int main() {
    setlocale(LC_ALL, "");
    fs::path path("C:/");
    vector<fs::path> b = getAllFilesFromDisk(path);
}

Tryied it out and it also throws same error (filesystem::filesystem_error). Unfortunately, doesn't work either :( — dummygum, May 08 '22 at 18:37
however, if i add try catch block it contains some of files, in my case from recycle bin but no more — dummygum, May 08 '22 at 18:44

How can i get all files from disk?

1 Answers1