0

Is it possible in Windows and Linux to mmap chunks of memory on a file that isn't as large as the mapping?

The file may contain only a few bytes worth of values, but I want to create multiple 1MB (in that ballpark) memory-mapping and read from this memory as the file grows. The file is written-to by the same program. I will never touch memory beyond the end of the file. But I don't want to redo the mmap calls with every new values that are appended into the file!

  • Linux: I checked man 2 mmap and it doesn't appear to forbid this kind of usage.
  • Windows: I checked the Windows API. The API devides the task into two functions, CreateFileMapping and MapViewOfFile. The latter is similar to mmap, and wants a handle that is returned by the former. But the size passed to MapViewOfFile is restricted to whatever size was given to CreateFileMapping! Can I simply call CreateFileMapping with whatever my max file size is going to be, let's say 100TB and Windows will let me create views of the file as the file grows till 100TB?

I checked folly's MemoryMap implementation, and it won't allow mappings larger than the file size. Neither does python's mmap. I haven't found a definitive answer for boost.interprocess.

Johannes Schaub - litb
  • 496,577
  • 130
  • 894
  • 1,212

2 Answers2

1

Subsequently I found an answer. I would like to quote my comments made to YangXiaoPo-MSFT's answer who was kind enough to provide me with first-hand MSFT information, after which I did some research on my own.

I subsequently discovered another answer that goes into some detail on how to resize an existing file mapping to be larger. It involves directly calling the Zw / Nt functions and not using the CreateFileMapping wrapper. As far as I know, CreateFileMapping creates a special "section" object that's backed by an arbitrary file - like there are sections .data for example that are too backed by files. Usually these sections are fixed size. But the sepecial Nt functions allow extending the section.

The mentioned answer about NtExtendSection is at https://stackoverflow.com/a/55068177/34509 .

I also uttered the following

Although I will probably not map the entire file into memory - it may grow muiltiple terabytes and even on 64bit, I'm unsure whether windows will find a large enough contiguous VM segment. I will create the file mapping over the entire file, and then map only individual chunks of the file.

Johannes Schaub - litb
  • 496,577
  • 130
  • 894
  • 1,212
0

The answer is intermediate in Windows. As CreateFileMapping states:

If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk and if the page protection allows write access (that is, the flProtect parameter specifies PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is increased to match the specified size of the file mapping object. If the file is extended, the contents of the file between the old end of the file and the new end of the file are not guaranteed to be zero; the behavior is defined by the file system. If the file on disk cannot be increased, CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL.

After a file mapping object is created, the size of the file must not exceed the size of the file mapping object; if it does, not all of the file contents are available for sharing.

YangXiaoPo-MSFT
  • 1,589
  • 1
  • 4
  • 22
  • Thanks for the answer, though I don't understand: The text doesn't unfortunately say what happens when the page protection doesn't allow write access. It says "and if the page protection allows write access" and describes the error condition when the file size cannot be increased because the disk is full. What happens when I map the file read-only and the mapping size is larger than the file is not specified. But I have a slight suspicion that another error would be thrown then. – Johannes Schaub - litb Jul 12 '23 at 10:45
  • I have a followup-question, if you allow. How can I update the size later on when the file grows? I.e regarding the second paragraph of your quote. I cannot close the mapping and open another one I suspect, because there may still be open views. Is there a possibility to increase the size in-place? – Johannes Schaub - litb Jul 12 '23 at 10:47
  • 1
    @Johannes Schaub - litb According to my test using [Obtaining a File Name From a File Handle](https://learn.microsoft.com/en-us/windows/win32/memory/obtaining-a-file-name-from-a-file-handle) example, `CreateFileMapping` can [create another new file mapping object with different size for the same file](https://i.stack.imgur.com/hqJJz.png) but that involves more questions to be verified. – YangXiaoPo-MSFT Jul 13 '23 at 01:33
  • @YangYang I subsequently discovered another answer that goes into some detail on how to resize an existing file mapping to be larger. It involves directly calling the Zw / Nt functions and not using the CreateFileMapping wrapper. As far as I know, CreateFileMapping creates a special "section" object that's backed by an arbitrary file - like there are sections .data for example that are too backed by files. Usually these sections are fixed size. But the sepecial Nt functions allow extending the section. – Johannes Schaub - litb Jul 13 '23 at 13:35
  • Check https://stackoverflow.com/a/55068177/34509 – Johannes Schaub - litb Jul 13 '23 at 13:36
  • Although I will probably not map the entire file into memory - it may grow muiltiple terabytes and even on 64bit, I'm unsure whether windows will find a large enough contiguous VM segment. I will create the file mapping over the entire file, and then map only individual chunks of the file. Do you think that would be a viable approach, together with above mentioned undocumented Nt functions to resize the section object? – Johannes Schaub - litb Jul 13 '23 at 13:38
  • You found the answer. Would you answer yourself? – YangXiaoPo-MSFT Jul 14 '23 at 02:58
  • Thanks again, I added an answer to that effect – Johannes Schaub - litb Jul 14 '23 at 10:12