6

Which is the best file system to run a web server and a database on debian ?

Example : On debian ext2 or ext3 take too long to manage huge folder with huge subfolder my administration task take to much time, i cant delete huge subfolders without waiting 30mins each.

Ophidian
  • 2,178
  • 14
  • 14
belaz
  • 595
  • 1
  • 5
  • 7
  • This is a question more suitable for http://serverfault.com/ or http://superuser.com/ I suppose. – akosma Jan 20 '10 at 13:47
  • Is a "huge subfolder" really such a common occurrence for web servers or database servers? For mail servers, yes, you need to think about it, but otherwise? – Peter Eisentraut Jan 20 '10 at 13:57
  • @Peter - yes: if you're storing many files on the filesystem, for example, instead of in a database (pros and cons of each discussed elsewhere) – warren Jan 20 '10 at 14:20
  • 1
    Consider storing your files in a hashed tree structure to minimize the number of files in any given folder. – Chris Nava Jan 20 '10 at 15:03

6 Answers6

3

Do your extX filesystems have dir_index enabled? (run tune2fs -l /dev/XXX to check) If not, try enabling that as a first step.

XFS handles massive directories well.

James
  • 7,643
  • 2
  • 24
  • 33
3

As James noted, ext{2,3} handles huge directories extremely well with the appropriate flags; But.... it sometimes doesn't feel like that.

Specifically:

  • A modern filesystem can do a very fast and scalable mapping from name to inode, meaning that it can (nearly) instantly open any given file no matter how big the directory it's in. Also answering any query (existence, permissions, size, owners, etc) about a specific path is largely unaffected by directory size.

But...

  • Any operation that works on the directory as a whole will have to linearly iterate all the files there, which can be really slow. For instance, ls by default sorts alphabetically the filenames; so it has to first read them all, then sort, then display, easily taking several minutes on multi-thousand-files directories. Another common issue is wildcard matching, which also has to read all existing filenames to return the matching subset.

conclusion: if you only use precisely-specified paths, any good filesystem will do. if you use wildcards, or frequently operate on the whole directory (listing, copying or deleting it), any filesystem will be too slow on huge directories.

Community
  • 1
Javier
  • 9,268
  • 2
  • 24
  • 24
  • My biggest knock against ext3 is its inefficient allocation compared to XFS or JFS -- especially with large files or lots of small files. When I've got a ton of storage space that I'm planning on using (big db, fileserver, etc), then I don't want to throw away a ton of space on metadata because ext3 is greedy. – Ophidian Jan 20 '10 at 15:21
0

First, check the wikipedia article on file systems: http://en.wikipedia.org/wiki/Comparison_of_file_systems.

Second, consider if you can reorganize your hierarchy to not have so many files at any given level.

Third, what kind of hardware are you running?

Fourth, this is probably a highly-opinion-oriented question, but I like ext3 or jfs, personally.

warren
  • 18,369
  • 23
  • 84
  • 135
0

I have always preferred XFS on systems that deal with large files. It will handle large high demand systems very well. It has great recovery tools. It also has a fast IO rate. As with anything there are drawbacks. Undeleting is tough. Creating and deleting directories takes longer then it does on other file systems.

Justin S
  • 350
  • 3
  • 15
0

This kind of depends on the details of "huge folder", specifically whether its a large number of small files or physically large files as well as how deep the tree of directories is.

XFS is a very solid file system that is excellent at working efficiently with large files. It often gets knocked in production environments for its aggressive caching of data in RAM and possible data loss due to sudden power failure (not file system corruption, just data loss) although pretty much every file system suffers from this same problem to some extent. The other gotcha is somewhat slower metadata operations when adding or deleting directories. This may be a deal-breaker for you if you have a deep directory tree, but I would suggest testing out XFS before dismissing it.

JFS is a rock solid file system noted for low CPU usage and well-rounded performance under many different loads. It's pretty much my go-to file system when I have a strong desire for the stability of ext3 but can't deal with the performance quirks (aka inefficient allocation and slow disk access) of the ext series of file systems. You may not find it quite as high performing with large files when compared to XFS.

Without further details on your targetted workload, I can't really give you a definitive suggestion, but I suspect JFS will be a very good choice if you don't have time for extensive tuning and benchmarking.

Ophidian
  • 2,178
  • 14
  • 14
  • unrelated anecdotic datapoint: when some of my filesystems grew beyond 8TB, i had to replace ext3 with something else. my 'default' option was XFS, but was scared by the data loss attitude (if you hardware fails, you lose data, no complains). so i tried a lot of 'pull the plug' scenarios. ext3 performed really well, losing in average less than one file per million on each hard failure. XFS wasn't far, with less than 5 lost files per million. JFS, on the other hand, lost a few hundred files per million _every time_. Of course, I went with XFS. – Javier Jan 20 '10 at 15:12
  • Interesting indeed. I've always thought most of the "data loss" issue with XFS is a myth perpetuated from its early days being integrated to the kernal (similar to the "ReiserFS will eat your filesystem" myths). I bypassed the whole matter by putting my XFS machines on a UPS. – Ophidian Jan 20 '10 at 15:18
0

For filesystems with a very large number of small files, ReiserFS has long been king. Ext3 is a good "all 'round" filesystem but ReiserFS really excels with large directories of small files. Until just recently it was still the default in Suse systems and is still a popular option there.

Scott Alan Miller
  • 497
  • 4
  • 7