16

I manage an application that contains a filestore in which all the files are stored with the filenames equal to their md5 sums. All files are stored in one directory. Currently there are thousands, but soon their should be millions of files on the server. The current server is running Ubuntu 11.10 on an ext4 filesystem.

Someone told me that it is not wise to put many files in a directory, as this will create significant increase in lookup time and reliability (he had a story about max files a single dir could point to, resulting in a big linked list). Instead he suggested to create sub directories with e.g. substrings of the filename. However, this will make some things in my application much more cumbersome.

Is this still true, or do modern filesystems (e.g. ext4) have more efficient ways to deal with this and naturally scale? Wikipedia has some details on filesystems, but it doesn't really say anything about max files per directory, or lookup times.

Jeroen Ooms
  • 2,239
  • 8
  • 34
  • 51

4 Answers4

8

The core of the problem is digging through the directory inode for the one file you want. Some filesystems do this better than others. Some scale close to the billions, but if you only have... 20K files getting to those files is markedly faster. Also, large file-counts create problems for certain tools and may make backup/restore a much harder problem as a result.

As it happens I ran into the exact same problem in our own development (md5sum as filename, scaling thereof). What I recommended to our developers is to chop the string into pieces. They went with groups of 4, but on the filesystem we were on at the time even that many would prove problematic from a performance perspective, so they ended up splitting on a group-of-3 for the first 6 triplets and leaving the rest as the filename in the terminal directory.

Group of 4: 4976/d70b/180c/6142/c617/d0c8/9d0b/bd2b.txt
Group of 3: 497/6d7/0b1/80c/614/2c6/17d0c89d0bbd2b.txt

This has the advantage of keeping directory sizes small, and since MD5sum is pretty random, it'll create balanced directory trees. That last directory is unlikely to ever get more than a few files. And wasn't that hard to work into our code. We work with multi-million file projects, so scaling was very important to us.

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
  • 4
    Just be cautious that if an attacker has the computational resources, he can deliberately create malicious data that will land in the same directory. An attacker with decent resources and today's technology could produce hashes that have the same first 9 hex digits (and thus collide in the first three directory levels) at a rate of about one every ten minutes. And, of course, full MD5 hashes can be generated today. – David Schwartz Dec 24 '11 at 04:12
7

The ext3 and later filesystems support hashed B-tree directory indexing. This scales very well as long as the only operations you do are add, delete, and access by name. However, I would still recommend breaking the directories down. Otherwise, you create a dangerous booby trap for tools (updatedb, ls, du, and so on) that perform other operations on directories that can blow up if the directory has too many entries.

David Schwartz
  • 31,449
  • 2
  • 55
  • 84
5

Modern filesystems handle very large directories very well, even to millions of files. But conventional tools do not. For example listing such a large directory with "ls" would take quite a long time since it would normally read the entire directory and sort it (although you can use ls -f to avoid sorting). It would not start showing files until all are read. Splitting the names helps in some cases, but not in all (for example rsync replication could still need to collect the entire tree of names).

Community
  • 1
Skaperen
  • 1,094
  • 2
  • 11
  • 23
-1

Might I suggest using an SQL database instead? This would likely transform this perceived weakness in your application into a strength.

RedScourge
  • 147
  • 5