4

If I have multiple threads generating blocks of a file, what is the best way to write out the blocks?

ex) 5 threads working on a file of 500 blocks, block 0 is not necessarily completed before block 1, but the output file on disk need to be in order. (block 0, block 1, block 2, .... block 499)

the program is in C++, can fwrite() somehow "random access" the file? the file is created from scratch, meaning when block 5 is completed, the file may still be of size 0 due to block 1~4 are not completed yet. Can I directly write out block 5? (with proper fseek)

This piece of code is performance critical, so I'm really curious about anything that can improve the perf. This looks like a multiple producer(block generators) and one consumer(output writer) scenario. The idea case is that thread A can continue generating the next block when it complete the previous.

if fwrite can be "random", then the output writer can simply takes outputs, seek, and then write. However not sure if this design can perform well in large scale.

Some limitations

  • Each block is of the same size, generated in memory
  • block size is known in advance, but not the total number of blocks.
  • the total size is a few GBs. Big.
  • There could be multiple jobs running on one server. each job is described at above. They have their own independent generators/writer, difference processes.
  • The server is a Linux/CentOS machine.
Solti
  • 633
  • 2
  • 6
  • 17
  • Are the blocks all the same size? – T33C May 10 '16 at 21:09
  • Why not just assemble the file in memory and the print it to disk once complete? – Baum mit Augen May 10 '16 at 21:09
  • `fseek` and ye shall find (though you probably want to accumulate a bare minimum of several kilobytes of consecutive data before doing a seek). – Jerry Coffin May 10 '16 at 21:12
  • You should be able to fseek + fwrite (you probably need to do both of these calls atomically if you want to parallelize it), but extend the file to the desired size before you do that. This is assuming that the final size of the file is known beforehand. :) – bialpio May 10 '16 at 21:14
  • Potentially also a good idea (if the blocks are big): Have every thread write its own file and the use a script to merge them afterwards. We do that in production actually. – Baum mit Augen May 10 '16 at 21:15
  • Yes, every block is of the same size and known in advance. However, the number of blocks is not known in advance. – Solti May 10 '16 at 21:18
  • The size of the file is multiple GB, and there could be multiple requests running simultaneously on one server. Therefore probably not safe to store the blocks in memory first and then flush out. – Solti May 10 '16 at 21:19
  • @Baum mit Augen your idea is interesting. probably I would like test and see if it performs better. the only thing is, I need to implement some logic to merge the sub-files to get the right order of blocks. this is do-able. – Solti May 10 '16 at 21:22
  • Re: `I need to implement some logic to merge the sub-files to get the right order of blocks` - or do you? Could you just leave them in multiple files and calculate the file name instead of offset? Fixes your locking issue, too. – Vlad Feinstein May 10 '16 at 21:27
  • I am thinking that the best way to go about doing this will be to use the OS API, what OS are you using? – T33C May 10 '16 at 21:29
  • 1
    Can't help you with Linux but I have found the standard libraries slower and less flexible than the OS API. Obviously, you sacrifice portability though. I recommend reading up on what the Linux API offers and giving that a go. Memory Mapped Files may be an easy and performant solution. – T33C May 10 '16 at 22:03
  • Start with the simplest code that works (write from a single thread, queue blocks from other threads for writing). Measure its performance. Make sure the writing to the file is the bottleneck. Define the goal: how fast do you need it to be. Update the question with the info. Click [edit]. – jfs May 10 '16 at 22:31

1 Answers1

1

Assuming each block is the same size, and that the blocks are generated in memory before they are required to be written to disk, then a combination of lseek and write would be perfectly fine.

If you are able to write the entire block in one write you would not gain any advantage in using fwrite -- so just use write directly -- however you would need some sort of locking access control (mutex) if all the threads are sharing the same fd -- since seek+write cannot be done atomically, and you would not want one thread to seek before an just before a second thread is about to write.

This further assume that your file system is a standard file system, and not of some exotic nature, since not all input/output device everything supports lseek (for example a pipe).

Update: lseek can seek beyond the end of file, just set the whence parameter = SEEK_SET and the offset to the absolute position in the file (fseek has the same option, but I have never used).

Scott Stensland
  • 26,870
  • 12
  • 93
  • 104
Soren
  • 14,402
  • 4
  • 41
  • 67
  • if I don't know how many blocks in total in advance, can fseek works? say if block 5 is the first one completed, the output file is still size 0, can it "fseek" to the block 5 position at that moment? – Solti May 10 '16 at 21:26
  • just google it. it seems fseek is of no problem exceeding EOF..!!! yeah~~ then this is a good solution candidate. – Solti May 10 '16 at 21:48
  • wondering if bringing Memory-Mapped-File for the output files would gain any extra performance? – Solti May 10 '16 at 21:53
  • You can seek beoynd the end-of-file and start writing there -- however Linux and Windows (FAT) behaves slightly different, since Linux supports files with hole, where Windows(FAT) write zeros to fill the holes -- making it inefficent. – Soren May 10 '16 at 22:33
  • I like memory mapped files for a lot of things, but they can be hard to tune for performing writes -- however in theory they should be the same performance as write(2) for most applications -- some people claim they should be faster as they require one less copy of the data (from user to kernel space) but I have never seen that to be a true benefit -- but your MMW – Soren May 10 '16 at 22:36
  • One significant benefit you have with memorymapped files is the issue of thread-safety between seek&write goes away -- however you are limited to files ~1 or 2Gb in size (forget the exact limit) – Soren May 10 '16 at 22:38
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/111579/discussion-between-solti-and-soren). – Solti May 10 '16 at 22:46
  • @Soren, Note that except for removable drives, NTFS is almost universally used now on Windows and does support file holes. – Dark Falcon May 10 '16 at 22:46