Create Large File in /dev/shm in parallel: performance

Question

I need to create a large file (several gbytes) in /dev/shm. which is a RAMdisk on Linux machines. This must be as fast as possible. I'm currently using 5 separate POSIX threads in C to create sequential 100 mbyte segments, then I have another thread which concatenates them onto the main output file.

This is fast, but I want to go faster. Can I eliminate the concatenate thread? Is there any way to open a file, and have each thread write it's 100 mbyte segment to the correct place in the final output file?

score 3 · Accepted Answer · answered Jun 07 '13 at 12:51

3

The fastest way to do this is to just call truncate() or ftruncate() to expand the file to the size you want.

You can then mmap() the file into the process's memory space, and have each thread write its section into the mapped area.

answered Jun 07 '13 at 12:51

caf

233,326
40
323
462

I tried this, and it might work. But I was able to 'create' files that were larger than the filesystem that contained them. Somehow, that doesn't seem right. – JB_User Jun 07 '13 at 12:56
So the basic idea is this: First, use truncate to create the huge file, then each thread uses lseek() to move forward to its spot in the file, then that thread writes the data there? Is that right? – JB_User Jun 07 '13 at 12:57
Okay. mmap(). Hadn't thought of this. – JB_User Jun 07 '13 at 12:58
`truncate` is not needed. `lseek` alone will do the job just fine. – R.. GitHub STOP HELPING ICE Jun 07 '13 at 13:05
Well, `truncate` is needed for the `mmap` approach, but not for `write`. – R.. GitHub STOP HELPING ICE Jun 07 '13 at 13:13

score 2 · Answer 2 · edited Jun 07 '13 at 22:09

Simply open the file once for each thread, lseek to the position the thread is suppose to start at, and write normally. Or, you could use pwrite to specify the location of the write; in that case, a single open could be shared by all the threads since they don't need to use the current file position.

If you want to ensure in advance that there's space available and reserved for the entire final file, posix_fallocate should be able to take care of that for you.

Create Large File in /dev/shm in parallel: performance

2 Answers2