Make calloc opportunistic

Question

On linux malloc behaves opportunistically, only backing virtual memory by real memory when it is first accessed. Would it be possible to modify calloc so that it also behaves this way (allocating and zeroing pages when they are first accessed)?

Note that for large chunks, `calloc` on Linux/glibc is already opportunistic (uses the same `MAP_PRIVATE|MAP_ANONYMOUS` mmap call, as newly-mmapped pages are promised to read as zeroes). — Anton Kovalenko, Apr 04 '15 at 20:49

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

2

It is not a feature of malloc() that makes it "opportunistic". It's a feature of the kernel with which malloc() has nothing to do whatsoever.

malloc() asks the kernel for a slap of memory everytime it needs more memory to fulfill a request, and it's the kernel that says "Yeah, sure, you have it" everytime without actually supplying memory. It is also the kernel that handles the subsequent page faults by supplying zero'ed memory pages. Note that any memory that the kernel supplies will already be zero'ed out due to safety considerations, so it is equally well suited for malloc() and for calloc().

That is, unless the calloc() implementation spoils this by unconditionally zeroing out the pages itself (generating the page faults that prompt the kernel to actually supply memory), it will have the same "opportunistic" behavior as malloc().

Update:

On my system, the following program successfully allocates 1 TiB (!) on a system with only 2 GiB of memory:

#include <stdlib.h>
#include <stdio.h>

int main() {
    size_t allocationCount = 1024, successfullAllocations = 0;
    char* allocations[allocationCount];
    for(int i = allocationCount; i--; ) {
        if((allocations[i] = calloc(1, 1024*1024*1024))) successfullAllocations++;
    }
    if(successfullAllocations == allocationCount) {
        printf("all %zd allocations were successfull\n", successfullAllocations);
    } else {
        printf("there were %zd failed allocations\n", allocationCount - successfullAllocations);
    }
}

I think, its safe to say that at least the calloc() implementation on my box behaves "opportunistically".

edited Jun 20 '20 at 09:12

Community

1
1

answered Apr 04 '15 at 20:55

cmaster - reinstate monica

38,891
9
62
106

"any memory that the kernel supplies will already be zero'ed". That's fine and dandy, but what about memory that is returned to the heap by `free` and then `calloc`ed again? – n. m. could be an AI Apr 04 '15 at 21:28
Thanks, I didn't know the kernel usually zeros all memory. What that doesn't explain is that I took an application (which I did not write) and changed calls to `calloc` to `malloc` and it crashed. I'm assuming this is because the allocated memory was not zeroed while the application assumed it would be. – chew socks Apr 04 '15 at 22:18
@n.m. Any memory that has not been returned to the kernel has to be zero'ed by `calloc()` if it is to be used for a new allocation. However, if the `free()` implementation returns memory to the kernel (via `munmap()` or `sbrk()`), a new mapping created via `mmap()` or `sbrk()` will again be zero'ed out by the kernel. – cmaster - reinstate monica Apr 05 '15 at 20:20
@chewsocks Requesting memory from the kernel and allocating memory are not the same thing. Memory requested from the kernel is used to back allocations via `malloc()`, but not every `malloc()` call requests memory from the kernel. Since `malloc()` is allowed to reuse memory from previously free'ed allocations, `malloc()` can return memory regions that have not been directly requested from the kernel, and which are not zero'ed consequently. Returning memory to the kernel is usually avoided by `malloc()` implementations to avoid the overhead of the syscalls. – cmaster - reinstate monica Apr 05 '15 at 20:30
@cmaster Thank! I understand now how `free()` doesn't always result in zeroed memory. Thanks for your example too. It's very similar to one I gave in this question http://stackoverflow.com/questions/29452164/why-does-allocating-large-chunks-of-memory-fail-when-reallocing-small-chunks-doe/29452188#29452188 maybe you could take a look at it? – chew socks Apr 05 '15 at 23:06

score 1 · Answer 2 · answered Apr 04 '15 at 20:50

From the related /proc/sys/vm/overcommit_memory section in proc:

The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes, even if it has not been "used" by them as of yet. A process which allocates 1GB of memory (using malloc(3) or similar), but only touches 300MB of that memory will only show up as using 300MB of memory even if it has the address space allocated for the entire 1GB. This 1GB is memory which has been "committed" to by the VM and can be used at any time by the allocating application. With strict overcommit enabled on the system (mode 2 /proc/sys/vm/overcommit_memory), allocations which would exceed the CommitLimit (detailed above) will not be permitted. This is useful if one needs to guarantee that processes will not fail due to lack of memory once that memory has been successfully allocated.

Though not explicitly said, I think similar here means calloc and realloc. So calloc already behaves opportunistically as malloc.

`calloc` is very different from `malloc`. `calloc` guarantees the memory it returns is initialized to zero. `malloc` does not. Unless you know that `calloc` manages to zero out pages without touching them, or somehow defers zeroing until first access, you should think twice before making such generalizations. — n. m. could be an AI, Apr 04 '15 at 21:26

Make calloc opportunistic

2 Answers2