1

Guys I am creating an application which will be saving a .txt file (approx 10 KB/file I guess) in a file system and will be saving its path in MS SQL database. Now as per my estimation, approximately 350-400 million files of this nature will need to be saved per year.

What is the best way or best file system to store these huge amounts of files ? Currently my servers run on Windows Server 2008 R2, can windows file system handle this much files ? or do I need to look into unix/linux solutions ?

Note : btw I only need to store them, they will be read once and then very rarely (in some special case) or maybe not at all. That is why I will be only keeping them for 2,3 years.

NewbieProgrammer
  • 119
  • 4
  • Normally with lots of small text files, I'd say "just store it all in the SQL database", but your max dataset is about 3.73TB pre year :-/ You could cut it up in monthly chunks (318GB), but I'm guessing the spec requires that the output is plain text? :-( – DutchUncle Jan 29 '15 at 13:11
  • 3
    Text compresses very well and putting groups of files into a .ZIP or other archive format instead of a sub directory reduces the filesystem file count. – Brian Jan 29 '15 at 13:18

1 Answers1

5

From a technical standpoint, NTFS can store about 4 billion files per filesystem.

Then again, with the amount of files you plan to store you need to think through your architecture more carefully; things like taking backups can take a needlessly long time with that many files, if done wrong.

Janne Pikkarainen
  • 31,852
  • 4
  • 58
  • 81
  • I read somewhere that in NTFS filesystem, there is index file limitation as index file gets large, it gets fragmented and then you can't add any more files. To avoid this you need to defrag index file regularly - increasing the complexity of the whole operation. :( – NewbieProgrammer Jan 29 '15 at 15:06
  • @NewbieProgrammer I asume this "index file" is the MFT. There are no significant limitations to the fragmentation of the MFT. – surfasb Jan 30 '15 at 22:58