write one big file or multiple small files

Question

I'm wondering what is better in therms of performance: write in one big text file (something about 10GB or more) or use a subfolder system that will have 3 levels with 256 folders in each one, the last level will be the text file. Example:

It will be heavy accessed (will be opened, append some text stuff then closed), so I don't know what is better, open and close file pointers thousand times in a second, or change a pointer inside one big file thousand times.

I'm at a core i7, 6GB DDR3 of RAM and a 60MB/s write disk speed under ext4.

What do you mean by heavy accessed? Read only or writes? How are you going to seek in a text file? Also, in most cases the IO is the slowest part of a program, so by making the file smaller (binary or compressed) you'll gain much speed. — ruslik, Oct 20 '11 at 22:48
@ruslik writes. I'll seek it by having a table of address in the beginning that simulates the folder structure. — Frederico Schardong, Oct 20 '11 at 22:54
@FredericoSchardong: And why it has to be text? What kind of data do you have? — ruslik, Oct 20 '11 at 22:55

score 2 · Accepted Answer · answered Oct 20 '11 at 22:49

You ask a fairly generic question, so the generic answer would be to go with the big file, access it and let the filesystem and its caches worry about optimizing access. Chances are they came up with a more advanced algorithm than you just did (no offence).

score 0 · Answer 2 · answered Oct 21 '11 at 06:54

To make a decision, you need to know answers to many questions, including:

How are you going to determine which of the many files to access for the information you are after?
When you need to append, will it be to the logical last file, or to the end of whatever file the information should have been found in?
How are you going to know where to look in any given file (large or small) for where the information is?
Your 256³ files (16 million or so if you use all of them) will require a fair amount of directory storage.

You actually don't mention anything about reading the file - which is odd.

If you're really doing write only access to the file or files, then a single file always opened with O_APPEND (or "a") will probably be best. If you are updating (as well as appending) information, then the you get into locking issues (concurrent access; who wins).

So, you have not included enough information in the question for anyone to give any definitive answer. If there is enough more information in the comments you've added, then you should have placed those comments into the question (edit the question; add the comment material).

write one big file or multiple small files

2 Answers2