0

I'm using a circa summer 2007 MacBook Pro (x86-64) with a 32KB L1 (I think), 4MB L2, and 4GB RAM; I'm running OS X 10.6.8.

I'm writing a standard radix sort in C++: it copies from one array to another and back again as it sorts (so the memory used is twice the size of the array). I watch it by printing a '.' every million entries moved.

If the array is at most 750 MB then these dots usually move quite fast; however if the array is larger then the whole process crawls to a halt. If I radix sort 512 MB in blocks and then attempt to merge sort the blocks, the first block goes fast and then again the process crawls to a halt. That is, my process only seems to be able to use 1.5 GB of RAM for the sort. What is odd is that I have 4 GB of physical RAM.

I tried just allocating an 8 GB array and walking through it writing each byte and printing a '.' every million bytes; it seems that everything starts to slow down around 1.5 GB and stays at that rate even past 4 GB when I know it must be going to disk; so the OS starts writing pages to disk around 1.5 GB.

I want to use my machine to sort large arrays. How do I tell my OS to give my process at least, say, 3.5 GB of RAM ? I tried using mlock(), but that just seems to slow things down even more. Ideas?

Nimantha
  • 6,405
  • 6
  • 28
  • 69
Daniel
  • 1,861
  • 1
  • 16
  • 24
  • The OS probably reserves a bit of every address to distinguish between user-space and kernel-space, so there is no possible way for a user process to use more than 2 GB out of 4 GB before virtual memory is invoked. – Kevin Grant Jul 22 '12 at 05:24
  • not on a 64-bit machine; certainly not for virtual addresses as I have malloc-ed huge arrays way bigger than 2 ** 32 and I have written a loop to write through an array larger than 2 ** 32, but once it got past 1.5 GB or so, it got rather slow as I was effectively writing straight to disk; and for physical addresses that would make even less sense, as if it's a 64-bit machine and can provide 64-bits to the user in the virtual space, what would be the point in using only 32-bit addresses in physical space? – Daniel Jul 22 '12 at 09:23
  • the problem is clearly that the kernel just doesn't want to allow one process to use more than 1.5 GB of physical RAM; I'm just wondering how to get it to back off on that restriction – Daniel Jul 22 '12 at 09:24

0 Answers0