How to distinguish out of memory versus out of address space in C on Linux?

Question

Suppose I'm running a piece of code on a 32 bit CPU and plenty of memory. And a process uses mmap to map a total of 2.8GB worth of file into it's address space. Then the process tries to allocate 500MB of memory using malloc. The allocation is bounded to fail and returns NULL due to not having enough address space; even though the system may have enough allocate-able memory.

The code looks something like this:

int main()
{
    int fd = open("some_2.8GB file",...);
    void* file_ptr = mmap(..., fd, ...);
    void* ptr = malloc(500*1024*1024);

    // malloc will fail because on 32bit Linux, a process can only have 3GB of address space
    assert(ptr == NULL);
    if(out_of_address_space())
        printf("You ran out of address space, but system still have free memory\n");
    else
        printf("Out of memory\n");
}

How could I detect the failure is caused by out of address space instead of allocate-able memory? Is out_of_address_space possible to implement?

Answer 1

How could I detect the failure is caused by out of address space instead of allocate-able memory?

You could calculate the amount of maximum virtual memory like bash does in ulimit -v - by querying getrlimit().

You can calculate the amount of "allocated" virtual memory by summing the differences between second and first column in /proc/pid/maps file.

Then the difference will give you the amount of "free" virtual space. You can compare that with the size you want to allocate and know if there is enough free virtual space.

Example: Let's compile a small program:

$ gcc -xc - <<EOF
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
int main() {
    void *p = malloc(1024 * 1024);
    printf("%ld %p\\n", (long)getpid(), p);
    sleep(100);
}
EOF

The program will allocate 1MB, print it's pid and address and sleep so we have time to do something. On my system if I limit virtual memory to 2.5M the allocation fails:

$ ( ulimit -v 2500; ./a.out; )
94895 (nil)

If I then sum the maps file:

$ sudo cat /proc/94895/maps | awk -F'[- ]' --non-decimal-data '{a=sprintf("%d", "0x"$1); b=sprintf("%d", "0x"$2); sum += b-a; } END{print sum/1024 " Kb"}'
2320 Kb

Knowing that the limit was set to 2500 Kb and the process is using 2320 Kb, there is only space to allocate 180 Kb, not more.

Possible C implementation for fun:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <stdbool.h>

size_t address_space_max(void) {
    struct rlimit l;
    if (getrlimit(RLIMIT_AS, &l) < 0) return -1;
    return l.rlim_cur;
}

size_t address_space_used(void) {
    const unsigned long long pid = getpid();
    const char fmt[] = "/proc/%llu/maps";
    const int flen = snprintf(NULL, 0, fmt, pid);
    char * const fname = malloc(flen + 1);
    if (fname == NULL) return -1;
    sprintf(fname, fmt, pid);
    FILE *f = fopen(fname, "r");
    free(fname);
    if (f == NULL) return -1;
    long long a, b;
    long long sum = 0;
    while (fscanf(f, "%llx-%llx%*[^\n]*", &a, &b) == 2) {
        sum += b - a;
    }
    fclose(f);
    return sum;
}

size_t address_space_free(void) {
    const size_t max = address_space_max();
    if (max == (size_t)-1) return -1;
    const size_t used = address_space_used();
    if (used == (size_t)-1) return -1;
    return max - used;
}

/**
 * Compares if there is enough address space for size
 */
bool out_of_address_space(size_t size) {
    return address_space_free() < size;
}

int main() {
    printf("%zu Kb\n", address_space_free()/1024);
    // ie. use:
    // if (out_of_address_space(500 * 1024 * 1024))
}

Answer 2

And a process uses mmap to map a total of 2.8GB worth of file into it's address space. Then the process tries to allocate 500MB of memory using malloc.

Don't mmap(2) the entire file at once !

Do mmap no more than one gigabyte (and no more than 2.5 gigabytes in total on a 32 bits Linux, including malloc-related mmap or sbrk). Then use mremap(2) and/or munmap(2). See also madvise(2). Be aware of the m modifier to fopen(3) mode string. In some cases, stdio(3) functions (using fseek and fread) might be enough and you could replace your mmap with them. Be aware of memory overcommitment and of the page cache. Both could be tunable thru /sys/ or /proc/ (see sysconf(3), sysfs(5), proc(5)...) and might be monitorable thru inotify(7) or userfaultfd(2) and/or signal(7).

Notice that malloc(3), dlopen(3) and shared libraries are also mmap-ing (thru ld.so(8)...) - and some malloc implementations are sometimes using sbrk(2) to manage small memory chunks. As an optimization, free(3) does not always munmap. Check with strace(1) and pmap(1) (or programmatically thru /proc/self/maps or /proc/self/status or /proc/self/statm, see proc(5)).

Some 32 bits Linux kernels could be specially configured (at their compile time) to accept slightly more than 3GBytes of virtual address space. I forgot the details. Ask on https://kernelnewbies.org/

Study the source code of your C standard library (e.g. GNU glibc). Most of them are open-source so you can improve them, e.g. musl-libc. You could use others (e.g. dietlibc), and you usually can redefine malloc. Budget a few months of efforts.

Read also Advanced Linux Programming (also here), Modern C, syscalls(2), the documentation of your C standard library, and a good operating system textbook.