I have a process where, after some time, a call to mprotect() fails with ENOMEM. The error is caused by "Internal kernel structures could not be allocated": at this point, the file /proc/<pid>/maps contains 65531 lines, which is of course suspiciously close to 2^16.
The process starts by taking a chunk of memory with mmap (1280MB, MAP_PRIVATE | MAP_ANONYMOUS), and calls mprotect() on single pages (or small numbers of pages) to enable and disable access to them, for debugging. The 1280MB mmaped region initially shows up as a single line in /proc/<pid>/maps, but each call to mprotect() can potentially "split" this region in three: the memory before, the modified page, and the memory after. After several changes, when two adjacent (but independently-modified) memory regions end up with the same access flags, they are usually merged again, so the total number of lines is kept reasonable.
But only "usually". In the case that fails, the /proc/<pid>/maps ends up with a large number of contiguous memory regions with the same protection, and I don't understand why they are not merged. In one-page examples, they are merged properly. When does it work differently and why?
For reference, the failing program is multi-threaded, although this should not have an effect (for each thread, the /proc/<tid>/maps file is identical). Seen on Linux kernels "2.6.35-30-generic #56-Ubuntu SMP" and "3.2.0-37-generic #58-Ubuntu SMP".
EDIT: how to reproduce:
hg clone https://bitbucket.org/pypy/stmgc
cd stmgc/c4
hg up d4e3aac8c458 # branch copy-over-original2
make debug-demo2
gdb ./debug-demo2 # lots and lots of colored output
It should fail at err = mprotect(..); assert(err == 0);
