How to remap view of shared memory via file mapping object?

Question

Say, if I have a shared file mapping object:

HANDLE hFileMapping = ::CreateFileMapping(INVALID_HANDLE_VALUE, &sa,
    PAGE_READWRITE, 0, 0x8000, L"MyObjectName");

and I got a small chunk of it for view, as such:

BYTE* pData = (BYTE*)::MapViewOfFile(hFileMapping, FILE_MAP_READ, 0, 0, 0x10);
if(!pData) return false;   //fail
DWORD dwcbFullSize = *(DWORD*)(pData + 0xC);

So then if I need to allocated more data, will it be acceptable to call MapViewOfFile again as such without first unmapping pData?

BYTE* pFullData = (BYTE*)::MapViewOfFile(hFileMapping,
      FILE_MAP_READ, 0, 0, dwcbFullSize);

PS. My goal is not to waste CPU cycles on mapping the entire 32K shared memory segment, when all I may need to read could be way less than that.

@RbMm: Yeah, thanks. That looks like it would be a cool function, had it been documented by MS. Just curious though, when `MapViewOfFile` maps shared memory, doesn't it load the whole page at once? Or does it do only as many bytes as its last parameter specified? — c00000fd, Oct 28 '17 at 23:39
@RbMm: So its first parameter is my `hFileMapping` and the second parameter is new mapped view size in bytes, and not the added size, correct? Also I'm assuming I will have to load it with `GetProcAddress` from `ntdll.dll`, right? — c00000fd, Oct 28 '17 at 23:43
@RbMm: OK. sorry, I'm struggling to understand you. Let's try with code. So you're saying that if I do `MapViewOfFile(hFileMapping, FILE_MAP_READ, 0, 0, 1)` that will have the same effect as if I did `MapViewOfFile(hFileMapping, FILE_MAP_READ, 0, 0, 0x10000)`. Is that right? — c00000fd, Oct 29 '17 at 00:17
@RbMm: I'm not asking about the offset, I'm asking about the size of the mapped byte array. — c00000fd, Oct 29 '17 at 00:32
>>will it be acceptable to call MapViewOfFile again as such without first unmapping pData? Yes, but you have to use the 3rd and 4th parameters of MapViewOfFile, and as RbMm suggested these are based on the memory allocation granularity of the system. Another option is to just map the same region and use an offset of the returned pointer. — kvr, Oct 29 '17 at 00:39
@RbMm that's why MS documentation says ''these are based on the memory allocation granularity of the system", so you have to use appropriate chunks. — kvr, Oct 29 '17 at 01:54

score 1 · Answer 1 · answered Oct 29 '17 at 01:31

for this task need use SEC_RESERVE attribute when we create file maping(section)

If the file mapping object is backed by the operating system paging file (the hfile parameter is INVALID_HANDLE_VALUE), specifies that when a view of the file is mapped into a process address space, the entire range of pages is reserved for later use by the process rather than committed. Reserved pages can be committed in subsequent calls to the VirtualAlloc function. After the pages are committed, they cannot be freed or decommitted with the VirtualFree function. This attribute has no effect for file mapping objects that are backed by executable image files or data files (the hfile parameter is a handle to a file). SEC_RESERVE cannot be combined with SEC_COMMIT.

so after create section with SEC_RESERVE attribute need once call MapViewOfFile - this call not reserve memory range but not commit pages. for commit pages need be call VirtualAlloc. finally we can free all memory by call UnmapViewOfFile

void DoDemo(ULONG cb)
{
    if (!cb)
    {
        return;
    }

    SYSTEM_INFO si;
    GetSystemInfo(&si);

    cb = (cb + (si.dwPageSize - 1)) & ~(si.dwPageSize - 1);

    if (HANDLE hFileMapping = CreateFileMapping(INVALID_HANDLE_VALUE, 0,
        PAGE_READWRITE|SEC_RESERVE, 0, cb, L"MyObjectName"))
    {
        // reserve address space with PAGE_READWRITE initial protection

        PVOID BaseAddress = MapViewOfFile(hFileMapping, 
            FILE_MAP_READ|FILE_MAP_WRITE, 0, 0, cb);

        // hFileMapping no more need
        CloseHandle(hFileMapping);

        if (BaseAddress)
        {
            // check, for test only
            ::MEMORY_BASIC_INFORMATION mbi;
            if (VirtualQuery(BaseAddress, &mbi, sizeof(mbi)) < sizeof(mbi) ||
                mbi.Type != MEM_MAPPED || mbi.State != MEM_RESERVE)
            {
                __debugbreak();
            }

            // map page by page
            PBYTE pb = (BYTE*)BaseAddress;
            do 
            {
                if (!VirtualAlloc(pb, si.dwPageSize, MEM_COMMIT, PAGE_READWRITE))
                {
                    GetLastError();
                    break;
                }
                *pb = '*';//write something
            } while (pb += si.dwPageSize, cb -= si.dwPageSize);

            //unmap all
            UnmapViewOfFile(BaseAddress);
        }
    }
}

however all this have sense do only for large by size sections. for 32kb(tiny size) the best will be just map all pages in single call

Thanks. My original mistake was that I didn't understand that `MapViewOfFile` deals with allocation of virtual memory. I wish they stressed that fact somewhere in the documentation. I explained my other findings in the [comments here](https://stackoverflow.com/a/47003615/843732). — c00000fd, Oct 29 '17 at 21:09
@c00000fd - usually it not deals with allocation of virtual memory. `SEC_RESERVE` on in memory only section special case — RbMm, Oct 29 '17 at 21:21

score 1 · Answer 2 · answered Oct 29 '17 at 18:16

There's a chance that I might have misunderstood the question, so please bear with me. I've decided it's easier to show what I was saying in the comments more clearly with some working code. @OP, perhaps from here you can further clarify if this does not address your question(s)?

So I've taken the sample code offered by MSFT and hacked together to demonstrate what I was saying (so the code is essentially MS example code).

https://msdn.microsoft.com/en-us/library/windows/desktop/aa366551(v=vs.85).aspx

You can create a VS Solution with 2 projects, here's the code for project/process A:

#include <windows.h>
#include <stdio.h>
#include <conio.h>
#include <tchar.h>

#define BUF_SIZE 256
TCHAR szName[] = TEXT("Global\\MyFileMappingObject");
TCHAR szMsg[] = TEXT("Message from first process.");

int _tmain()
{
    HANDLE hMapFile;
    LPCTSTR pBuf;

    hMapFile = CreateFileMapping(
        INVALID_HANDLE_VALUE,    // use paging file
        NULL,                    // default security
        PAGE_READWRITE,          // read/write access
        0,                       // maximum object size (high-order DWORD)
        BUF_SIZE,                // maximum object size (low-order DWORD)
        szName);                 // name of mapping object

    if (hMapFile == NULL)
    {
        _tprintf(TEXT("Could not create file mapping object (%d).\n"),
            GetLastError());
        return 1;
    }
    pBuf = (LPTSTR)MapViewOfFile(hMapFile,   // handle to map object
        FILE_MAP_ALL_ACCESS, // read/write permission
        0,
        0,
        BUF_SIZE);

    if (pBuf == NULL)
    {
        _tprintf(TEXT("Could not map view of file (%d).\n"),
            GetLastError());

        CloseHandle(hMapFile);

        return 1;
    }


    CopyMemory((PVOID)pBuf, szMsg, (_tcslen(szMsg) * sizeof(TCHAR)));
    _getch();

    UnmapViewOfFile(pBuf);

    CloseHandle(hMapFile);

    return 0;
}

Now here's some modified code for the second project/process B. See this for reference: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366548(v=vs.85).aspx

#include <windows.h>
#include <stdio.h>
#include <conio.h>
#include <tchar.h>
#pragma comment(lib, "user32.lib")

#define BUF_SIZE 256
TCHAR szName[] = TEXT("Global\\MyFileMappingObject");

int mapDataAtOffset(DWORD offset, size_t bytesToRead, LPVOID* outData, LPVOID* outMapAddress);

int _tmain()
{
    LPCTSTR pBuf;
    LPVOID outMapAddress = nullptr;
    LPVOID outMapAddress2 = nullptr;

    auto ret = mapDataAtOffset(0, 8, (LPVOID*)&pBuf, &outMapAddress);

    MessageBox(NULL, pBuf, TEXT("Process2"), MB_OK);

    ret = mapDataAtOffset(8, 8, (LPVOID*)&pBuf, &outMapAddress2);

    MessageBox(NULL, pBuf, TEXT("Process2"), MB_OK);

    if(outMapAddress)UnmapViewOfFile(outMapAddress);
    if (outMapAddress2)UnmapViewOfFile(outMapAddress2);

    return 0;
}

int mapDataAtOffset(DWORD offset, size_t bytesToRead, LPVOID* outData, LPVOID* outMapAddress) {
    HANDLE hMapFile;      // handle for the file's memory-mapped region
    BOOL bFlag;           // a result holder
    DWORD dBytesWritten;  // number of bytes written
    DWORD dwFileMapSize;  // size of the file mapping
    DWORD dwMapViewSize;  // the size of the view
    DWORD dwFileMapStart; // where to start the file map view
    DWORD dwSysGran;      // system allocation granularity
    SYSTEM_INFO SysInfo;  // system information; used to get granularity
    LPVOID lpMapAddress;  // pointer to the base address of the
                          // memory-mapped region
    char * pData;         // pointer to the data
    int i;                // loop counter
    int iData;            // on success contains the first int of data
    int iViewDelta;       // the offset into the view where the data
                          //shows up

    DWORD FILE_MAP_START = offset;

    // Get the system allocation granularity.
    GetSystemInfo(&SysInfo);
    dwSysGran = SysInfo.dwAllocationGranularity;

    // Now calculate a few variables. Calculate the file offsets as
    // 64-bit values, and then get the low-order 32 bits for the
    // function calls.

    // To calculate where to start the file mapping, round down the
    // offset of the data into the file to the nearest multiple of the
    // system allocation granularity.
    dwFileMapStart = (FILE_MAP_START / dwSysGran) * dwSysGran;
    _tprintf(TEXT("The file map view starts at %ld bytes into the file.\n"),
        dwFileMapStart);

    // Calculate the size of the file mapping view.
    dwMapViewSize = (FILE_MAP_START % dwSysGran) + bytesToRead;
    _tprintf(TEXT("The file map view is %ld bytes large.\n"),
        dwMapViewSize);

    // How large will the file mapping object be?
    dwFileMapSize = FILE_MAP_START + bytesToRead;
    _tprintf(TEXT("The file mapping object is %ld bytes large.\n"),
        dwFileMapSize);

    // The data of interest isn't at the beginning of the
    // view, so determine how far into the view to set the pointer.
    iViewDelta = FILE_MAP_START - dwFileMapStart;
    _tprintf(TEXT("The data is %d bytes into the view.\n"),
        iViewDelta);

    hMapFile = OpenFileMapping(
        FILE_MAP_ALL_ACCESS,   // read/write access
        FALSE,                 // do not inherit the name
        szName);               // name of mapping object

    if (hMapFile == NULL)
    {
        _tprintf(TEXT("Could not open file mapping object (%d).\n"),
            GetLastError());
        return 1;
    }

    // Map the view and test the results.

    lpMapAddress = MapViewOfFile(hMapFile,            // handle to
                                                      // mapping object
        FILE_MAP_ALL_ACCESS, // read/write
        0,                   // high-order 32
                             // bits of file
                             // offset
        dwFileMapStart,      // low-order 32
                             // bits of file
                             // offset
        dwMapViewSize);      // number of bytes
                             // to map
    if (lpMapAddress == NULL)
    {
        _tprintf(TEXT("lpMapAddress is NULL: last error: %d\n"), GetLastError());
        return 3;
    }

    // Calculate the pointer to the data.
    pData = (char *)lpMapAddress + iViewDelta;
    *outData = pData;
    *outMapAddress = lpMapAddress;

    CloseHandle(hMapFile); // close the file mapping object, doesn't matter as long as view is still mapped
    return 0;
}

So to answer your question

will it be acceptable to call MapViewOfFile again as such without first unmapping pData?

The example above achieves that. Unless I have misunderstood your query, in which case if you can clarify it'd be great.

Thanks for the code sample. I did some tests as well. It turns out that I didn't fully understand how `MapViewOfFile` worked. I thought that it would give me "direct access" to the shared memory of the file-mapping object. In that case if I already assigned 16 bytes for viewing, why not let me view the next 32 bytes, etc. But that obviously would not be compatible with the memory protection scheme in Windows NT, where each process has its own isolated set of memory pages. So `MapViewOfFile` cannot give me "direct" access to the shared memory, like Windows 9x would've done... — c00000fd, Oct 29 '17 at 20:53
What happens when you call `MapViewOfFile` is that it maps N number of virtual memory pages to the shared memory of the file-mapping object. Since each memory page is 4K bytes, and VM is aligned on 64K bytes in RAM, the offset must be divisible by 64K, and its size by 4K. This means two things for my example if I call `MapViewOfFile` for 1 byte and then call it again for 2 bytes. — c00000fd, Oct 29 '17 at 20:58
1) The first call will return the whole page of virtual memory, i.e. 4K bytes assigned to address A. and 2) The second call to `MapViewOfFile` will also return the entire VM page of 4K bytes, assigned to addess B. And `A != B` — c00000fd, Oct 29 '17 at 20:58
Keep in mind it's limited to the process address space (32-bit vs 64-bit). Depending on how large of a range you are mapping into view, you could potentially run out of address space on a 32-bit process. This happened to one of our apps, and we instead chose to deploy 64-bit only. — kvr, Oct 29 '17 at 21:02
I also understand now why MS don't have an API to extend the view of an already mapped shared memory. Because of this tie to the virtual memory, if the first call returns, say, one page of VM, or 4K bytes. Since the memory is also mapped to its VM address, there's no guarantee that a consecutive address of the next VM block will be available at the next address when you request to extend the memory. That is why there's no API to do it. (Well, aside from `reserving` it, which doesn't deal with assigning of physical memory address.) — c00000fd, Oct 29 '17 at 21:02
Yes, obviously a 32-bit process is very constrained in how much virtual memory it can allocate. I think it is capped at 2GB on paper, but in reality if you're able to allocate a contiguous block of more than 1GB you may consider yourself very lucky. So 64-bit process is the way to go. (However painful it may be for the purposes of IPC, and by providing two builds of the app: 32-bit and 64-bit for 32-bit OSes.) — c00000fd, Oct 29 '17 at 21:06
in this way problem (?) that `lpMapAddress` is arbitrary and not related to `dwFileMapStart`. so you not got **contiguous** memory. every chuck will be mapped at random address. faster of all this is unacceptable. and every chunk need unmap separate. by using `MapViewOfFileEx` we can use concrete address for make map contiguous but not possible 100% garantee that selected memory range will be free — RbMm, Oct 29 '17 at 21:20

How to remap view of shared memory via file mapping object?

2 Answers2