Allocating more than 4GB memory in a 64 bit system

Question

I'm running this code, compiled on 64 bits vc++ 2005, on Windows Server 2008 R2 with 32GB. There is an access violation inside the for loop.

#include <iostream>
using namespace std;


int main(int argc, char* argv[])
{   
    double *x = new double[536870912];

    cout << "memory allocated" << endl;

    for(long int i = 0; i < 536870912; i++)
    {   
        cout << i << endl;
        x[i] = 0;
    }

    delete [] x;
    return 0;
}

So if there is no exception in new double[536870912], why am I getting an access violation when doing an assignment over a particular array position?

Another point worth mentioning is that this program was succesfully tested on another computer.

One problem is that (I think) `long int` is 32 bits on 64-bit Windows, so the loop will never terminate. You should change the type of `i` to `size_t` to be sure it's big enough for any array index. I've no idea whether that's the only problem, though. — Mike Seymour, Jul 30 '13 at 16:44
The thing is that it prints out over a hundred numbers and then there is an access violation, so it seems it is not reaching the limit. — Jordi, Jul 30 '13 at 16:46
Just because the computer has sufficient memory, doesn't mean it can find a free `4GB` consecutive block in there. — Kirk Backus, Jul 30 '13 at 16:46
@MikeSeymour Even so, a little over 500 million shouldn't be a problem even for a 32-bit integer. — Some programmer dude, Jul 30 '13 at 16:48
@Mike Seymour The loop iteration size is much smaller than `2^32` though. — Mark B, Jul 30 '13 at 16:48
@JoachimPileborg: True; I lost count of the digits and thought there were 5G of them. That's not the culprit then. — Mike Seymour, Jul 30 '13 at 16:48
@KirkBackus true, but if `new[]` cannot allocate the requested amount then it should throw an exception at the allocation point. — zindorsky, Jul 30 '13 at 16:49
`536870912 * sizeof(double)` yields 0 for me. Looks like a wrap around inside `operator new`. — jrok, Jul 30 '13 at 16:50
@KirkBackus new doesn't retun null unless using the nothrow version which isn't the case here? — stijn, Jul 30 '13 at 16:50
@aaronman not here it isn't. This is the nothrow syntax: `double *x = new (std::nothrow) double [536870912]`. — zindorsky, Jul 30 '13 at 16:53
You may have compiled on 64-bit windows, but the compiler you are using is in 64-bit mode, right? — Neil Kirk, Jul 30 '13 at 16:57
What is "the particular array position"? What compiler have you used? There is nothing wrong with the program. You have the live repro; attach a debugger and investigate. — James McNellis, Jul 30 '13 at 16:58
If the allocation fails and you are not using the no throwing version, it does NOT return null. It doesn't return anything. An exception is thrown. Checking for null does nothing. — Neil Kirk, Jul 30 '13 at 16:59
@JamesMcNellis Visual C++ 2005, and it fails on iteration 768 — Jordi, Jul 30 '13 at 17:00
Did you try rebooting the windows machine? Sounds like a joke, I know... but... windows... you never know. — DanielKO, Jul 30 '13 at 17:00
@user4272 Rather than the array index, I think it would be more interesting to know the culprit address (ie. `&x[i]`). Wouldn't surprise me if it was on a page boundary. — syam, Jul 30 '13 at 17:10
@syam Iterations and addresses: 764-0000000000F17F60 765-0000000000F17F68 766-0000000000F17F70 767-0000000000F17F78 768-0000000000F17F80 — Jordi, Jul 30 '13 at 17:13
@stijn: ever heard of overcommiting memory allocations? Memory fragmentation? Kernel in an unstable state? I know Macs may need a reboot every week or so if you need a large contiguous block of memory, otherwise the allocation fails. — DanielKO, Jul 30 '13 at 17:13
@FlorisVelleman I have tried both. In debug it reaches the 768 iteration and in release it reaches the 486 iteration. — Jordi, Jul 30 '13 at 17:18
@user4272: is the system running under a virtual machine that might do memory overcommitment? That means you would detect failures only when actually writing to the memory. — DanielKO, Jul 30 '13 at 17:19
For what it's worth, this program works fine in VS2012 on Win8 x64 — mark, Jul 30 '13 at 17:19
@DanielKO yes to all. And I seriously doubt it has anything to do with this. And even if it were, it's not exactly a proper solution, nor probably the solution the OP is looking for. — stijn, Jul 30 '13 at 17:19
What is the smallest value you can allocate before it crashes? 4GB+1? — Neil Kirk, Jul 30 '13 at 17:46
It might be interesting to know that it works just fine if you pull out the constant into a variable (at least for me). My compiler also warns about a constant overflow when just using the constant directly, so I suspect that the number being used isn't the number you expect. — Chris Cooper, Jul 30 '13 at 17:49
@stijn The code contains no errors, so the issue must therefore be in the system. How is asking about known problems about the system in question not helping? — DanielKO, Jul 30 '13 at 18:30
what happens if you malloc the memory (this one will return null for failure) — Neil Kirk, Jul 30 '13 at 19:10

Adrian Maire · Answer 1 · 2013-08-01T10:28:06.103

It is probably one of the following problem:

long int is 32-bits: that mean your maximum value is 2147483647, and sizeof(double)*536870912 >= 2147483647. (I don't really know if that has sense. It probably depend on how the compiller work)
Your allocation is failing.

I suggest you to test the following code:

#include<conio.h>
#include <iostream>
using namespace std;

#define MYTYPE unsigned long long


int main(int argc, char* argv[])
{   
    // Test compiling mode
    if (sizeof(void*) == 8) cout << "Compiling 64-bits" << endl;
    else cout << "Compiling 32-bits" << endl;

    // Test the size of mytype
    cout << "Sizeof:" << sizeof(MYTYPE) << endl;
    MYTYPE len;

    // Get the number of <<doubles>> to allocate
    cout << "How many doubles do you want?" << endl;
    cin >> len;
    double *x = new (std::nothrow) double[len];
    // Test allocation
    if (NULL==x)
    {
        cout << "unable to allocate" << endl;
        return 0;
    }
    cout << "memory allocated" << endl;

    // Set all values to 0
    for(MYTYPE i = 0; i < len; i++)
    {   
        if (i%100000==0) cout << i << endl;
        x[i] = 0;
    }

    // Wait before release, to test memory usage
    cout << "Press <Enter> key to continue...";
    getch();

    // Free memory.
    delete [] x;

}

Editing: Using this code, i just achieved allocate a single block of 9GB.

Allocating more than 4GB memory in a 64 bit system

1 Answers1

Linked