3

I have a call to calloc for 1 element of just over 1 gigabyte. This call returns NULL, and checking errno reveals an insufficient memory error. However, during testing I have almost 4 gigabytes of free RAM, not to mention available virtual memory.

After researching the issue, the only thing I can come up with for why calloc fails is that I do not have enough contiguous memory available. However, it seems to me that if I have 4Gb available memory, it should be easy enough to have ~= 1Gb contiguous memory. Is there a way to check for contiguous memory availability? If this is the problem, is there a straightforward way to "defragment" the memory?

Or does this have something to do with the block size of allocated memory, so that it is actually trying to allocate way more memory than I have available? What are my options for determining cause of failure? What are my options for successfully allocating this much memory?

I am on a native Windows system with 12Gb of memory. Next I will try running in a VM, and giving the VM several gigabytes of memory, and checking if the virtual machine can non-transparently access enough contiguous memory. I will post the results here if I complete that today.

alk
  • 69,737
  • 10
  • 105
  • 255
taz
  • 1,506
  • 3
  • 15
  • 26
  • Your windows is 32-bit, right? – Alexander Putilin Aug 24 '12 at 23:54
  • 3
    You need to read this: http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx. You are running out of *address space*, which has nothing to do with RAM whatsoever. – Todd Li Aug 24 '12 at 23:57
  • @Nya No. 64-bit windows (although target platform might be 32). – taz Aug 24 '12 at 23:57
  • OK, I changed the tag for you – Gorpik Aug 24 '12 at 23:58
  • I think it's highly unlikely you'll be able to get a handle to one full gigabyte of contiguous memory under most environments, the best you can do is look at things your specific environment offers (e.g. [low fragmentation heaps](http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx)) or using a memory mapped file with a backing 1GB file. – ta.speot.is Aug 24 '12 at 23:58
  • @pst There isn't a magic number involved. It's the size of certain variables plus the a memory size setting. – taz Aug 25 '12 at 00:00
  • 1
    If you have to target the 32bit platform, avoid relying on 1GB+ contiguous address space. It's hard to defragment the address space given that there is only 2GB-3GB available to you. Do some bookkeeping and allocate the memory in smaller chunks. If you can, target 64bit platform to avoid all such hassles. – Todd Li Aug 25 '12 at 00:04
  • Ok, I'm naive and I am targeting a 32-bit platform. Should have thought of this before. – taz Aug 25 '12 at 00:09

1 Answers1

6

One important key to making that succeed is to have 1GB of virtual address space available to your application. With 32-bit Windows applications, the default situation is that an application starts out with essentially 2GB of address space.

If you do very many allocations in your application prior to the one that is failing, it is quite possible that you have carved out enough of the original address space so as to not leave 1GB of continuous address space left. An application could perform just two small "strategically placed" allocations that would break up the address space such that the remaining portions are all less than 1GB.

nobody
  • 19,814
  • 17
  • 56
  • 77
Mark Wilkins
  • 40,729
  • 5
  • 57
  • 110
  • So how can I check for this? Other than failing a call to calloc()? – taz Aug 25 '12 at 00:10
  • I don't know how to "check" for it. If you really need a 1GB chunk and there is no other way around it, perhaps you should allocate it very first thing. If it doesn't succeed then, it probably is not going to. I would move to a 64-bit platform, though. – Mark Wilkins Aug 25 '12 at 00:36
  • 4
    Use the SysInternals utility `vmmap` (http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx) to view the virtual memory map of a Windows process. Be sure to enable the menu option to view free memory, and sort by size and type. – Mark Tolonen Aug 25 '12 at 01:12
  • @taz: I work on a system that commits ~1GB (on 32-bit systems anyway) of contiguous memory up front and then uses it as needed. We back off a little if we can't allocate the entire 1GB and don't run into any problems. We have to do it up front though or risk a failure when it's too late to do anything sensible in the face of one. – Ed S. Aug 25 '12 at 06:30
  • 1
    Am confused: if you have 2GiB of virtual address space (b/c Windows reserves the other 2GiB), wouldn't you only need one pathological allocation right in the middle? – geometrian Aug 25 '12 at 06:30
  • @IanMallett What is a "pathological allocation"? – taz Aug 27 '12 at 14:58
  • @tax: If the available address space for a user application is 0 - 2^31, then an allocation of a few bytes at the middle of that (2^30), would effectively cut the address space in two halves both less than 1GB. And the actual address space starts at higher than 0, so the real total (by default) would be less than 2GB to start with. – Mark Wilkins Aug 27 '12 at 15:10