Using mmap with hint address is the address of a global variable and MAP_FIXED

Question

I have 2 or more processes accessing to a shared memory. I want to create a global variable in each process and then mapping the address of this variable to the shared memory using mmap API with MAP_FIXED flag. So that, when reading/ writing to this shared memory, I just need to access to the global variable ( the same way as we share global variable between threads, but here I would like to shared global variable between processes).

I define the global variable in each process as below:

typedef struct     // This struct define the shared memory area
{
   int data1;
   int data2;
   // ...
} SharedMemory;

// the following attribute (supported by GCC) make the start address of the variable aligned to 4KB (PAGE_SIZE)
 __attribute__((aligned(4096))) SharedMemory gstSharedMemory; // shared global variable
 int giOtherVar = 10;                                              // Another normal global variable

Then using mmap to map the shared memory to this global variable:

void* lpShmAddr = mmap(&gstSharedMemory,
                        sizeof(gstSharedMemory),
                        PROT_READ | PROT_WRITE,
                        MAP_SHARED | MAP_FIXED,
                        iFd,                        // File descriptor to the shared memory
                        0);

However, if sizeof(gstSharedMemory) is not multiple of PAGE_SIZE (4kb), and because the OS will always round up the map size to multiple of page size, all the bytes in the rounded-up region are initialized to 0. And it may cause the data of other global variable (for example: giOtherVar) to become Zero if their address is within this rounded-up region.

To overcome this situation, I use a byte array to backup the rounded-up region and recover it as below:

unsigned char byBkupShm[PAGE_SIZE]    =  { 0 } ;    
memcpy(&gbyBkupShm[0], 
      ((unsigned char*)&gstSharedMemory+ sizeof(gstSharedMemory)), 
       PAGE_SIZE - (sizeof(gstSharedMemory)% PAGE_SIZE));
void* lpShmAddr = mmap(&gstSharedMemory,
                        sizeof(gstSharedMemory),
                        PROT_READ | PROT_WRITE,
                        MAP_SHARED | MAP_FIXED,
                        iFd,                        // File descriptor to the shared memory
                        0);
 memcpy( ((unsigned char*)&gstSharedMemory+ sizeof(gstSharedMemory)), 
          &byBkupShm[0],
          PAGE_SIZE - (sizeof(gstSharedMemory)% PAGE_SIZE));

And finally, I access to shared memory like this:

// Write to shared memory:
gstSharedMemory.data1 = 5;

// Read from shared memory;
printf("%d", gstSharedMemory.data1);

My question is: Is there any potential problem with this implementation?

Editted: Thank to @None and his idea, I define a macro as below to make my struct aligned and rounded up to PAGE_SIZE, but at the same time, still provide the actual size of the struct if I need:

#define PAGE_SIZE       (4 * 1024)                                  // Page Size: 4KB
#define SHM_REG          __attribute__((aligned(PAGE_SIZE)))        // Aligned to 4KB boundary

#define DEFINE_SHM( structName_, shmSizeVar_, structContent_)       \
typedef struct SHM_REG structContent_ structName_;                  \
int shmSizeVar_ = sizeof(struct structContent_);  



// Using

DEFINE_SHM(
    MySharedMemory,                 // Struct Name of shared memory
    giSizeOfMySharedMemory,         // Global Variable 
    {
        int a;
        int b;
        char c;
    }
);
    
printf("Rounded Size: %d\n", sizeof(MySharedMemory));  // = 4096
printf("Acutal Size: %d\n", giSizeOfMySharedMemory);   // = 12 

Answer 1

Make sure the shared memory structure is both aligned to and sized as a multiple of page size:

#include <stdlib.h>

#ifndef  PAGE_SIZE
#define  PAGE_SIZE  4096
#endif

typedef struct __attribute__((aligned (PAGE_SIZE))) {
    /*
     * All shared memory members
    */
} SharedMemory;

At run time, before mapping the shared memory, verify it first:

SharedMemory  blob;

    if (PAGE_SIZE != sysconf(_SC_PAGESIZE)) {
        ABORT("program compiled for a different page size");
    } else
    if (sizeof blob % PAGE_SIZE) {
        ABORT("blob is not sized properly");
    } else
    if ((uintptr_t)(&blob) % PAGE_SIZE) {
        ABORT("blob is not aligned properly");
    } else
    if (MAP_FAILED == mmap(...)) {
        ABORT("could not map shared memory over blob");
    }

While this is a hack, at least this way it is safe, in Linux.

Answer 2

Yes, the potential problem is that giOtherVar is now shared as well, because the entire page is shared.

The normal way to do this is to not use MAP_FIXED, and let mmap choose a location for you, and store the pointer. You say you cannot do this because it would be a massive code change.

There's probably a way to use a linker script to force gstSharedMemory to be on a page by itself, but linker scripts are tricky.

You could add 4096 bytes of padding to SharedMemory so it's always bigger than 4096 bytes, and then not share the last page (which could overlap other global variables).

You could add an unused 4096 byte array after gstSharedMemory and hope that the compiler will put it right after gstSharedMemory.

You could make the next variable also 4096-byte aligned and hope the compiler doesn't decide to put other variables in the gap.

... or you could just use the pointer design, then #define gstSharedMemory (*gpstSharedMemory) so that you don't have to change all your code.

Answer 3

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <unistd.h>

typedef struct shared_data_s {
        int a;
        int b;
} __attribute__((aligned(4096))) shared_data_t;


shared_data_t g_data;
shared_data_t *g_data_ptr;

void child()
{
        printf("child set g_data to 1/2\n");
        g_data.a = 1;
        g_data.b = 2;
        sleep(2);
        printf("child visit g_data a=%d, b=%d\n", g_data.a, g_data.b);
}

void parent()
{
        sleep(1);
        printf("parent visit g_data a=%d, b=%d\n", g_data.a, g_data.b);
        g_data.a = 10;
        g_data.b = 20;
        sleep(3);
}

int main(int argc, char *argv[])
{

        g_data_ptr = mmap(&g_data, sizeof(g_data), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
        printf("%s: size=%d, &g_data=%p, g_data_ptr=%p\n", __func__, sizeof(g_data), &g_data, g_data_ptr);

        pid_t pid;
        pid = fork();
        if (pid == 0) {
                child();
        } else {
                parent();
        }

        return 0;
}