I have an image-processing set of tools that may end up consuming incredibly large amounts of memory because it is used to process images that are massive (~50k x 50k x 1000 -> 2.5 terapixel). While I do keep things on disk as much as possible and processing individual slices (so down to 2.5 gigapixel) when I can, sometimes it just doesn't work.
What I would like to be able to do is have a chunk of memory that I can write to that, if necessary, will be written to disk. This is the basic idea behind swap space, but doesn't require root/admin access and a restart to setup. This swap space would be specific to my process and to the memory I specify.
One way to do this would be use memory-mapped files. However this has a few disadvantages I believe. It is possible that memory-mapped files would work, and maybe I just need to have these misunderstandings corrected. Please inform me how memory-mapping actually works (preferably on Windows and Linux).
Do memory-mapped files try to sync to disk when able? I would like them to NEVER sync unless necessary for RAM purposes (given that I never call msync / FlushViewOfFile).
If I first only write page-size chunks of data to a memory-mapped region, is the underlying file ever read? On Linux: can the underlying file be a sparse file created with ftruncate-ing to the desired size?
On Linux: is there a way to get a private non-copy-on-write memory-map?
Does changing the memory protection (mprotect / VirtualProtect) cause data to be synchronized?
