C/C++ - Memory map file using mmap

Question

I'm new to memory mapping files and am a bit confused. Is it possible to map files larger than the total amount of memory because as I understood memory mapping uses demand paging and will hold only the currently relevant pages in memory. Is this right or will my application crash when using more space than the actual memory.

Thanks

EDIT OS: Ubuntu 14.04 LTS x86_64 App-Bitness: 64bit (I guess: pointers are 8 byte)

I'm trying to allocate memory from the mapped file in order to store the tree inside the mapped file.

#define MEMORY_SIZE 300000000

unsigned char *mem_buffer;
void *start_ptr;

void *my_malloc(int size) {
    unsigned char *ptr = mem_buffer;
    mem_buffer += size;

    return ptr;
}

void *my_calloc(int size, int object_size) {
    unsigned char *ptr = mem_buffer;
    mem_buffer += (size * object_size);

    return ptr;
}

void init(const char *file_path) {
    int fd = open(file_path, O_RDWR, S_IREAD | S_IWRITE);

    if (fd < 0) {
        perror("Could not open file for memory mapping");
        exit(1);
    }

    start_ptr = mmap(NULL, MEMORY_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
    mem_buffer = (unsigned char *) start_ptr;

    if (mem_buffer == MAP_FAILED) {
        perror("Could not memory map file");
        exit(1);
    }

    printf("Successfully mapped file.\n");
}

void unmap() {
    if (munmap(start_ptr, MEMORY_SIZE) < 0) {
        perror("Could not unmap file");
        exit(1);
    }

    printf("Successfully unmapped file.\n");
}

main method:

int main(int argc, char **argv) {

    init(argv[1]);

    unsigned char *arr = (unsigned char *) my_malloc(6);
    arr[0] = 'H';
    arr[1] = 'E';
    arr[2] = 'L';
    arr[3] = 'L';
    arr[4] = 'O';
    arr[5] = '\0';

    unsigned char *arr2 = (unsigned char *) my_malloc(5);
    arr2[0] = 'M';
    arr2[1] = 'I';
    arr2[2] = 'A';
    arr2[3] = 'U';
    arr2[4] = '\0';

    printf("Memory mapped string1: %s\n", arr);
    printf("Memory mapped string2: %s\n", arr2);

    struct my_btree_node *root = NULL;

    insert(&root, arr, 10);
    insert(&root, arr2, 20);

    print_tree(root, 0, false);

//  cin.ignore();

    unmap();

    return EXIT_SUCCESS;
}

@Malkocoglu I moved the question here `http://stackoverflow.com/questions/29213142/c-memory-map-a-b-tree` because the questions asked here is answered. — aQuip, Mar 23 '15 at 14:58

Malkocoglu · Answer 1 · 2015-03-23T13:21:30.443

4

It is possible (and very normal) to map files larger than the total amount of memory but your app must be able to address it. You have a limited address range if your system/app is 32-bit. Your system & app must be 64-bit to address bigger than several gigabyte sized files...

Note that: Even if your file is small, you may still not be able to map it. Maybe you consumed your address space by simply allocating memory for other purposes (stack space for threads, memory for application buffers/lists, etc.)

edited Mar 23 '15 at 13:21

answered Mar 23 '15 at 13:02

Malkocoglu

2,522
2
26
32

1

Actually it is 4GB but in practice 2GB or 3GB depending addressing mode. – Hynek -Pichi- Vychodil Mar 23 '15 at 13:12
Do I have to use `mmap64` in order to use the whole 64bit or is `mmap` sufficient? – aQuip Mar 23 '15 at 13:19
@Malkocoglu The problem now is that I get the error `Cannot allocate memory`if I try to map bigger size than the available memory. Do I have to manually unmap and then map with another offset to handle a big file? – aQuip Mar 23 '15 at 13:54
@aQuip: What is your system & app bitness ? If it is 32-bit, this error is most probably normal. Tell your bitness and show us some code ! – Malkocoglu Mar 23 '15 at 14:06

score 3 · Accepted Answer · answered Mar 23 '15 at 13:15

Here's an excerpt from The GNU C Library: Memory-mapped I/O

Since mmapped pages can be stored back to their file when physical memory is low, it is possible to mmap files orders of magnitude larger than both the physical memory and swap space. The only limit is address space. The theoretical limit is 4GB on a 32-bit machine - however, the actual limit will be smaller since some areas will be reserved for other purposes.

C/C++ - Memory map file using mmap

2 Answers2

Linked