6

I'm creating a utility in C++ to be run on Linux which can convert videos to a proprietary format. The video frames are very large (up to 16 megapixels), and we need to be able to seek directly to exact frame numbers, so our file format uses libz to compress each frame individually, and append the compressed data onto a file. Once all frames are finished being written, a journal which includes meta data for each frame (including their file offsets and sizes) is written to the end of the file.

I'm currently using ifstream and ofstream to do the file i/o, but I am looking to optimize as much as possible. I've heard that mmap() can increase performance in a lot of cases, and I'm wondering if mine is one of them. Our files will be in the tens to hundreds of gigabytes, and although writing will always be done sequentially, random access reads should be done in constant time. Any thoughts as to whether I should investigate this further, and if so does anyone have any tips for things to look out for?

Thanks!

rcv
  • 6,078
  • 9
  • 43
  • 63
  • How random are the reads? Is the range unbounded or bounded? – MSN Apr 20 '10 at 23:53
  • The reads are actually not too random, as common use will entail a user seeking to a particular frame, and then playing quite a few in sequence. Sorry - not sure what you mean about the range? – rcv Apr 20 '10 at 23:59
  • 1
    On a side note - that does not seem to be a good way to store video - even if you need lossless quality. – slacker Apr 21 '10 at 00:11
  • 1
    Slacker: I agree in most cases, but we need to have the ability to seek to exact frame numbers (not just the closest I-frame or any such approximation which is the best ffmpeg seems to be able to do with any of its formats). If there is a fast library/format out there that does exactly what I need, then I would love to hear about it! – rcv Apr 21 '10 at 00:15
  • 2
    The format is called M-JPEG, and ffmpeg supports it. It is essentially a sequence of independently compressed JPEG frames. – slacker Apr 21 '10 at 00:37
  • Ahh, but I need lossless quality - in fact I need to store pixel data up to 64bits/channel, so I don't know of any format (even MNG, which is limited to 16 bits/channel) that can handle this. – rcv Apr 21 '10 at 01:37
  • @Boatzart: JPEG 2000 includes a good lossless mode, but the 64 bits/channel requirement is really a tough one. May I know what is this going to be used for? – slacker Apr 21 '10 at 01:59
  • @slacker I work in a computational neuroscience/vision lab (http://ilab.usc.edu), and we represent the outputs of various neural network channels as well as our video frames in the same template'd "Image" class. All of our I/O modules must support both of these seamlessly, hence the need for high precision. I need the exact seeking because I'm working on a ground truthing video annotation tool for very high resolution footage which allows users to create keyframed animations to denote object bounding polygons over time... I guess niche would be an understatement :) Thanks for all of the help – rcv Apr 21 '10 at 03:21

2 Answers2

10

On a 32-bit machine your process is limited to 2-3 GB of user address space. This means that (allowing for other memory use) you won't be able to map more than ~1 GB of your file at a time. This does NOT mean that you cannot use mmap() for very large files - just that you need to map only part of the file at a time.

That being said, mmap() can still be a large win for large files. The most significant advantage is that you don't waste memory for keeping the data TWICE - one copy in the system cache, one copy in a private buffer of your application - and CPU time to make those copies. It can be an even more major speedup for random access - but the "random" part must be limited in range to your current mapping(s).

slacker
  • 2,142
  • 11
  • 10
  • Right, the approach I was thinking was to mmap as large as the architecture allows when the user reads a frame, and then just read from that map for as long the data I need is in there. Does anyone know how the overhead of a mmap() call compares to a seek()? – rcv Apr 21 '10 at 00:12
  • @Boatzart: They're comparable. The main cost here is the kernel call and the mapping done by kernel. But that STILL needs to be done for `seek()`. – slacker Apr 21 '10 at 00:26
  • 1
    I'd suggest mapping smaller parts of the file at a time - so that you don't waste too much memory and address space. The overhead of an `mmap()` call is negligible compared to the time needed to handle data on the order of megabytes. – slacker Apr 21 '10 at 00:32
  • 1
    Note that you can have multiple mappings of a single file simultaneously. – slacker Apr 21 '10 at 00:34
7

If your files are 10 GB or more, then don't even think about trying to use mmap() on a 32-bit architecture. Go directly to a 64-bit OS, which should be able to handle it just fine.

Note that files that are mapped into memory space don't actually consume the same amount of RAM (as the file size), so you won't need to install hundreds of gigabytes of RAM in your machine.

Greg Hewgill
  • 951,095
  • 183
  • 1,149
  • 1,285