13

The address space for 64 bit addressing is absolutely huge. I have a program that will mmap several chunks of memory, each of the order of 100 - 500 MB. I will inevitably be remapping a few times, which may cause some fragmentation of available contiguous space.

Whatever fragmentation of space occurs, it is surely going to be trivially small concerned the available address space.

My question is: given these constraints, under normal circumstances, can I expect all mmap requests to succeed (i.e. not fail due to fragmentation)? What reasons might there be for them failing?

I am aware that the heap doesn't have the whole space to itself, but I imagine it has the vast majority.

Mac OS / Linux.

Joe
  • 46,419
  • 33
  • 155
  • 245

4 Answers4

29

Be aware that the address space size for "64bit operating systems" is not necessarily covering the full 64bit range.

On x64 (64bit x86, aka 'AMD64'), for example, the actual available virtual address range is "only" 2x128TB, i.e. 48bit in two disjoint 47bit chunks. On some SPARC systems, it's 2x2TB, or 2x8TB (41/44 bits). This is due to the way the MMU works on these platforms.

In addition to these architectural limitations, the way the operating system lays out your address spaces also plays a role here.
64bit Windows on x64, for example, limits the virtual address size of an (even 64bit) application to 8TB (each kernel and user side1).

On UN*X systems (including Linux, MacOSX and the *BSDs), there is RLIMIT_AS that one can query for via getrlimit(), and - within system-specific limits - adjust via setrlimit. The ulimit command uses those. They return/set the upper bound, though, i.e. the allowed total virtual address space, including all mappings that a process can create (via mmap() or the malloc backend, sbrk()). But the total address space size is different from the maximum size of a single mapping ...

Given this, it's not that hard to exhaust the virtual address space even on 64bit Linux; just try, for a test, to mmap() the same 500GB file, say, two hundred times. The mmap will fail, eventually.

In short:

  • mmap() will definitely fail once you're out of virtual address space. Which is, somewhat surprisingly, achievable on many "64bit" architectures simply because virtual addresses might have less than 64 significant bits. The exact cutoff depends on your CPU and your operating system, and an upper bound can be queried via getrlimit(RLIMIT_AS) or set via setrlimit(RLIMIT_AS).
  • Address space fragmentation can occur on 64bit by frequently using mmap() / munmap() in different orders and with different-sized blocks. This will ultimately limit the maximum size you'll be able to map as a single chunk. Predicting when exactly this will happen is hard since it both depends on your "mapping history" and the operating systems' virtual address space allocation algorithm. If ASLR (address space layout randomization) is done by the OS, it might be unpredictable in detail, and not exactly reproducible.
  • malloc() will also fail no later than reaching the total VA limit, on systems (like Linux) where overcommit allows you to "ask" for more memory than there is in the system (physical + swap).
  • On machines / operating systems where no overcommit is enabled, both malloc() and mmap() with MAP_ANON and/or MAP_PRIVATE will fail when physical + swap is exhausted, because these types of mappings require backing store by actual memory or swap.

Little technical update: Like x86 and sparc mentioned above, the new ARMv8 (64bit ARM, called "AArch64" in Linux) MMU also has a "split" address space / an address space hole - out of the 64 bits in an address, only 40 are relevant. Linux gives 39bit for user, virtual addresses 0 ... onwards, 39bit for kernel, virtual addresses ... 0xFFFFFFFF.FFFFFFFF, so the limit there is 512GB (minus what's in use already at the time the app tries the mmap).
See comments here (from the AArch64 architecture enabler kernel patch series).

FrankH.
  • 17,675
  • 3
  • 44
  • 63
  • 2
    +1 your rep should be more than 815 with nice answers like this. – R.. GitHub STOP HELPING ICE Feb 07 '11 at 18:33
  • About the fragmentation.. I tried what you said with 32 files all in the size between 1 and 4 GB, mapping and unmapping them in loops on a 32bit system. Apart from one file that was 4.0GB, mmap always reused the same address for each file even though I passed NULL as the parameter for the requested pointer. Is the fragmentation only on 64bit with 500GB files visible? – Marenz May 05 '11 at 16:38
  • 1
    to create address space fragmentation via `mmap`, perform map/unmap operations in different order and with different sizes. I.e. say mmap 1000 files a 1MB each, unmap every 3rd of that, mmap 150 files of 2MB each, unmap a random 20 of that, ... this sort of thing. A simple loop of mmap/munmap on the same data set will not fragment (unless a very buggy ASLR algorithm is in use). It depends on the ASLR implementation whether a mmap/munmap/mmap will re-use the same address or not (it might, but if you start the same program again under a different PID the address chosen may differ). – FrankH. May 09 '11 at 09:48
8

mmap and malloc may fail due to lack of physical memory (RAM + swap). In general, the Unix standard speaks of address ranges that are "allowed for the address space of a process". Apart from that,

If a function be advertised to return an error code in the event of difficulties, thou shalt check for that code, yea, even though the checks triple the size of thy code and produce aches in thy typing fingers, for if thou thinkest ``it cannot happen to me'', the gods shall surely punish thee for thy arrogance.

(Spencer)

Fred Foo
  • 355,277
  • 75
  • 744
  • 836
0

Just check the list of errors in man mmap.

Maxim Egorushkin
  • 131,725
  • 17
  • 180
  • 271
  • Yes, but of all of those except `ENOMEM` (and one or two others) are avoidable. I'm asking if'/why something like `ENOMEM` might happen. – Joe Feb 07 '11 at 15:50
  • 2
    There can be a limit on the amount of virtual memory a process can reserve. See `ulimit -v`. – Maxim Egorushkin Feb 07 '11 at 16:07
0

Well, the OS / C library is free to decide whether there is enough memory or not. There's really no way for you to know how it decides this (it's obviously OS-specific, possibly even tunable), but it's possible that at some point the OS says "no more".

Possible reasons for failure to allocate memory:

  • Linux will overcommit memory, but this is tunable. Depending on how it is configured, you may get ENOMEM as soon as you exceed RAM+Swap.
  • If memory limits have been set using ulimit, you may get ENOMEM even sooner
sleske
  • 81,358
  • 34
  • 189
  • 227
  • Any properly-configured Linux system for robust server, workstation, or realtime use will have overcommit disabled. Running Linux with overcommit enabled is a lot like running Windows 95... – R.. GitHub STOP HELPING ICE Feb 07 '11 at 17:39
  • @R: Well, as far as I recall, overcommit is turned on by default under Linux, so I guess a good many distributions will ship this as the default config, and thus quite a few servers will run it. It can be problematic, of course. – sleske Feb 07 '11 at 17:43
  • I know it's on by default. But if I were hiring Linux admins, making sure they know what overcommit is and how to turn it off would be one of my first interview questions. :-) – R.. GitHub STOP HELPING ICE Feb 07 '11 at 18:33