c++ boost Get files from Disk in Parallel

Question

I'm looking for a fast way to get a List of Files with certain attributes, in parallel from Disk.

Attributes: file size, absolute file path

Currently i'm using boost filesystem and a recursive call with directory iterators. It's fine for small datasets but for a Million files in say 50.000 Folders its not great.

Usage Environment: OS: FreeBSD, Linux, Windows Filesystems: ZFS, ext4, NTFS

Basic Idea:

Thread Pool
SubTreeWalker Object
Partition root folder among threads
subtreewalker asks threadpool for each new dir in subdir if there are lazy threads
if 4 == true, assign directory to subtreewalker object in lazy thread.

What do you think of the basic idea, is it sound? Are there any implications of parallel access to the B+ Tree of the filesystem?

Parallel disk access sounds like it'd be slower than just reading the files sequentially. — twentylemon, Jun 23 '15 at 20:34
Most data into and out of a hard disk is serial (i.e. SATA interface). So what would you gain by opening more than one file in parallel? — Thomas Matthews, Jun 23 '15 at 21:14

score 1 · Answer 1 · answered Jun 24 '15 at 08:27

There are two common bugs in directory scanning. You propose the second (threading), probably because you're suffering from the first. That's doing a depth-first search. Directories are pretty contiguous on disk. Doing a breadth-first search takes advantage of this. Doing a depth-first search leads to much slower scans as directories are repeatedly visited.

score 0 · Answer 2 · answered Jun 23 '15 at 20:34

This may perform worse than a linear search since you will create more random-access reads on the disk by hammering it with several threads. If you want to speed up processing of the files (I assume you want to do something with them, not just look at them), I suggest creating a thread that scans the directories, queue the found files in a list and use one or more secondary threads to pop files of one by one of the queue and process them. This saves the wait before you finish scanning.

Re-arranging the file structure may also speed up the process, if possible. One million files in 500,000 directories sounds extremely ineffecient.

Edited to 50.000 Folders (typo) – eul3r Jun 23 '15 at 20:55 — eul3r, Jun 23 '15 at 20:55

c++ boost Get files from Disk in Parallel

2 Answers2