Writing many large files quickly in C++

Question

I have a program which gets a stream of raw data from different cameras and writes it to disk. The program runs these sorts of recordings for ~2 minutes and then another program is used to process the frames.

Each raw frame is 2MB and the frame rate is 30fps (ie. data rate is around 60MB/s) and I'm writing to an SSD which can easily handle a sustained > 150MB/s (tested by copying 4000 2MB files from another disk which took 38 seconds and Process Explorer shows constant IO activity).

My issue is that occasionally calls to fopen(), fwrite() and fclose() stall for up to 5 seconds which means that 300MB of frames build up in memory as a back log, and after a few of these delays I hit the 4GB limit of a 32 bit process. (When the delay happens, Process Explorer shows a gap in IO activity)

There is a thread which runs a loop calling this function for every new frame which gets added to a queue:

writeFrame(char* data, size_t dataSize, char* filepath)
{
    // Time block 2
    FILE* pFile = NULL;
    fopen(&pFile, filepath, "wb");
    // End Time block 2

    // Time block 3
    fwrite(data,1,dataSize,pFile);
    // End Time block 3

    // Time block 4
    fclose(pFile);
    // End Time block 4
}

(There's error checking too in the actual code but it makes no difference to this issue) I'm logging the time it takes for each of the blocks and the total time it takes to run the function and I get results which most of the time look like this: (times in ms)

TotalT,5,       FOpenT,1,       FWriteT,2,    FCloseT,2
TotalT,4,       FOpenT,1,       FWriteT,1,    FCloseT,2
TotalT,5,       FOpenT,1,       FWriteT,2,    FCloseT,2

ie. ~5ms to run the whole functions, ~1ms to open the file, ~2ms to call write and ~2ms to close the file.

Occasionally however (on average about 1 in every 50 frames, but sometimes it can be thousands of frames between this problem occurring), I get frames which take over 4000ms:

TotalT,4032,    FOpenT,4023,    FWriteT,6,    FCloseT,3

and

TotalT,1533,    FOpenT,1,       FWriteT,2,    FCloseT,1530

All the frames are the same size and its never fwrite that takes the extra time, always fopen or fclose

No other process is reading/writing to/from this SSD (confirmed with Process Monitor).

Does anyone know what could be causing this issue and/or any way of avoiding/mitigating this problem?

Each frame is being written to a separate file? Maybe you could write groups of frames to a single file and have a separate thread post-processing them to take a file of N frames and segment into separate files afterwards (to alleviate the fopen/fclose calls in the video processing thread). This would then allow your writeFrame func to run much quicker and not slow down the video capture process. — spartygw, Jan 17 '14 at 01:10
consider checking that `fopen` and `fclose` are actually opening and closing successfully, also check how many threads are running, it may be that you are hitting the maximum thread amount. — Matthew Pigram, Jan 17 '14 at 01:11
You don't mention the platform or the filesystem type used on the ssd. If it is windows this would be a good place to use asynchronous writes. — Dan, Jan 17 '14 at 01:18
@spartygw yeah, its writing each frame to a separate file. I'm considering writing to a massive file and dicing it up later but this still should be possible, right? :( — digitalPhonix, Jan 17 '14 at 01:27
@MatthewPigram Sorry, I wasn't clear, there are only two threads. One waits for start/stop commands, a thread which runs this function in a loop and the frames are delivered to a queue in a driver callback — digitalPhonix, Jan 17 '14 at 01:28
@Dan Running on windows. Wouldn't rapid async writes to different files be bad? — digitalPhonix, Jan 17 '14 at 01:28
Each frame a separate file? That's very likely the problem. You are creating 30*60*2=3600 files in 2 minutes. While sequential writes is fast, create file/close file is not, as they are not sequential access to the disk, they need to access directory structure/file table entry, granted a lot of are in memory cache but you'll hit disk occasionally. Another problem with ntfs is the directory is not scalable, the more files in the same directory, the slower it is to create a new file (evidently this only shows up when the number of files in one directory goes to more than 10K). — X.J, Jan 17 '14 at 01:33
Its NTFS (just thinking now... would FAT be better for this? I'm not going to hit the file size limit). How does windows file copy sustain much higher data rates writing the same files? — digitalPhonix, Jan 17 '14 at 01:54
@X.J.: SSDs care about non-sequential access? (_edit: derp, write caching, nm)_ — Lightness Races in Orbit, Jan 17 '14 at 02:16
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683%28v=vs.85%29.aspx — Dan, Jan 17 '14 at 12:51

score 1 · Answer 1 · answered Jan 17 '14 at 09:30

I'm going to side with X.J., you're probably writing too many files to a single directory. A solution could be to create a new directory for each batch of frames. Also consider calling SetEndOfFile directly after creating the file, as that will help Windows allocate sufficient space in a single operation.

FAT isn't a real solution as it's doing even worse on large directories.

score 1 · Answer 2 · answered Jan 17 '14 at 09:41

Prepare empty files (2 MB files filled with zeros) So that space is already "ready", then just overwrite these files. Or create a file that is a batch of several frames, so you can reduce number of files.

there are libraries for doing compression and decompression and playback of videos:

libTheora may be usefull because already compress frames (well you will need to output the video in a single file) and do that pretty fast (lossy compression by the way).

Writing many large files quickly in C++

2 Answers2