3

Is it possible for a single process running a 32-bit compiled version of python in Snow Leopard (64-bit machine) to appear to consume > 4GB (say 5.4GB) of virtual memory as seen by the top command?

I did a file ...python to see that the binary was not x86, yet it appeared to be consuming over 5GB of memory.

My guess is that the libraries that were used (RPy) were 'mmap'ing chunks of data, and the in-memory cache was appearing under the memory footprint of my process.

Or maybe I haven't verified that the Python binaries were 32-bit. Or maybe there's some 32-bit/64-bit commingling going (libffi?).

Totally confused.

phuclv
  • 37,963
  • 15
  • 156
  • 475
Jag the Reducer
  • 51
  • 4

2 Answers2

2

No, it's physically impossible. That doesn't stop the OS assigning more than it can use due to alignment and fragmentation, say, it could have a whole page and not actually map in all of it. However it's impossible to actually use over 4GB for any process, and most likely substantially less than that for kernel space.

Puppy
  • 144,682
  • 38
  • 256
  • 465
  • The limit is most of the time slightly over 3GB and the author of the program needs to specificaly ask for the ability to use that much memory (the default limit is 2GB). This applies to Windows, don't know about Linux. – PeterK Jul 27 '10 at 20:03
  • @peterK: The exact limit isn't important. What is important is that it's far south of 5.4 GB. – Puppy Jul 27 '10 at 22:31
  • i know, just wanted to add some more info. – PeterK Jul 28 '10 at 06:24
  • hmm... I seem to be hitting this limit on ubuntu 12.04 with PAE... doh! – monkut Jun 12 '13 at 14:18
1

It is possible if the processes is using some kind of insane long/far/extended pointers and mapping data into and outof a 32-bit address space as it needs it, but at that point it hardly qualifies as 32-bit anyway. (Python most definitely does not do this, so @DeadMG's answer is almost certainly what is actually happening.)

David X
  • 3,998
  • 3
  • 29
  • 32
  • 1
    Premiere CS4 uses another approach, that is loading another process for each 4GB of memory, so that the whole program can access as much as existing physical memory – phuclv Sep 02 '13 at 05:34
  • @LưuVĩnhPhúc, if you did that you'd have one process with 2.6gb and another with 2.8gb (or something along those lines), not one process with 5.4gb. (That is a good way to deal with address space constaints, but it's not really aplicable here.) – David X Sep 02 '13 at 23:42
  • of course that's 2 (or more) seperate processes but they're using some process interconnection to transfer date between each other. Anyway, Premiere cs5 and above drop support for 32-bit environment – phuclv Sep 02 '13 at 23:58
  • Note that hardware segmentation doesn't help on x86. `seg_base + off` results in a linear *32-bit* virtual address (which is then mapped by hardware to physical). The mapping in/out scheme you describe would have to be done by software. e.g. `mmap(MAP_FIXED|MAP_SHARED)` against a huge file could persist your large working set in the pagecache and map portions of it into your virtual address space. – Peter Cordes Apr 10 '20 at 16:57