6

I'm writing a MATLAB program that will generate a matrix with 1 million rows and an unknown amount of columns (at max 1 million).

I tried pre-allocating this matrix:

a=zeros(1000000,1000000)

but I received the error:

"Maximum variable size allowed by the program is exceeded."

I have a feeling that not pre-allocating this matrix will seriously slow the code down.

This made me curious: What is the maximum array size in MATLAB?

Update: I'm going to look into sparse matrices, because the result I am aiming for in this particular problem will be a matrix consisting for the larger part of zeros.

chappjc
  • 30,359
  • 6
  • 75
  • 132
Pieter
  • 647
  • 4
  • 9
  • 16
  • 2
    One has to wonder why your code requires such a large matrix? Perhaps you would be better off with sparse matrices? – CookieOfFortune Feb 25 '10 at 07:48
  • Well... if you imagine it, you're allocating 1,000,000,000,000 variables. if you were representing bits with that, that would be about a terabit... which is huge. You're probably representing a 32-bit integer, so thats ~32 terabits ._. thats... huge... I honestly doubt you have 4000 gigabytes of ram. – Warty Feb 25 '10 at 07:49
  • 3
    This won't answer your question, but it may help reframe the problem. How much available memory do you have? If these are doubles they take 8 bytes each. This requires 10^12 * 8B = 8TB. Where would you store it? If your matrix can be assumed to be sparse (i.e. comprising mostly 0s), you can use Matlab's `sparse` command to generate a sparse matrix. – Nathan Fellman Feb 25 '10 at 07:50
  • Note that preallocating sparse matrices is almost never a good idea. You should never build up a sparse matrix incrementally, adding elements to it one at a time. –  Feb 25 '10 at 11:03

4 Answers4

8

Take a look at this page, it lists the maximum sizes: Max sizes

It looks to be on the order of a few hundred million. Note that the matrix you're trying to create here is: 10e6 * 10e6 = 10e12 elements. This is many orders of magnitude greater than the max sizes provided and you also do not have that much RAM on your system.

My suggestion is to look into a different algorithm for what you are trying to accomplish.

Community
  • 1
  • 1
CookieOfFortune
  • 13,836
  • 8
  • 42
  • 58
7

To find out the real maximum array size (Windows only), use the command user = memory. user.maxPossibleArrayBytes shows how many bytes of contiguous RAM are free. Divide that by the number of bytes per element of your array (8 for doubles) and you know the max number of elements you can preallocate.

Note that as woodchips said, Matlab may have to copy your array (if you pass by value to a subfunction, for example). In my experience 75% of the max possible array is usually available multiple times.

Community
  • 1
  • 1
Jonas
  • 74,690
  • 10
  • 137
  • 177
  • 5
    I agree with the above, just wanted to point out that MATLAB uses a copy-on-write mechanism when passing matrices to functions, meaning that unless that function changes the input its as if it were passed by reference. – Amro Feb 25 '10 at 17:18
2

The Limits

There are two different limits to be aware of:

  1. Maximum array size (in terms of number of elements) allowed by MATLAB, regardless of current memory availability.
  2. Current bytes available for a single array -- the (current) maximum possible array size in bytes.

The first limit is what causes "Maximum variable size allowed by the program is exceeded", not the second limit. However the second one is also a practical limit of which you must be aware!

Checking the Limits

The maximum number of elements allowed for an array is checked as follows:

>> [~,maxsize] = computer
maxsize =
   2.8147e+14

According to the documentation for the computer command, this returns:

maximum number of elements allowed in a matrix on this version of MATLAB

This is a static MATLAB limit on number of elements, not affected by the state of the computer (hardware specs and current memory usage). And at over 2 petabytes for a double array of that length, it's also way higher than any computer of which I am aware!

On the other hand, the largest practical array size that you can create at any given moment can be checked by the memory command:

>> memory
Maximum possible array:     35237 MB (3.695e+10 bytes) *
Memory available for all arrays:     35237 MB (3.695e+10 bytes) *
Memory used by MATLAB:      9545 MB (1.001e+10 bytes)
Physical Memory (RAM):     24574 MB (2.577e+10 bytes)

*  Limited by System Memory (physical + swap file) available.

As the message says, these values are based on actual current memory availability, taking into account both physical memory and the swap file (collectively, virtual memory).

If needed, these values can accessed programmatically by m = memory;.

Adjusting the Limits

The first limit (the hard limit) has been fixed up until R2015a, where it can now be changed (but only reduced to a fraction of system memory) through the following setting:

enter image description here

You can't increase it beyond your system limits.

The second limit obviously has no "setting" in MATLAB since it's based on available memory and computer configuration. Aside from adding RAM, there's not a lot you can do: (1) pack to consolidate workspace memory and perform "garbage collection", but this may only help on certain platforms, and (2) increasing page file size to allow other stuff to swap out and give MATLAB more physical memory. But be cautious when relying on your page file as your computer may become unresponsive if page file thrashing happens.

chappjc
  • 30,359
  • 6
  • 75
  • 132
0

In older versions of Matlab that don't include the memory command, you can use:

feature memstats

Physical Memory (RAM):
    In Use:                              738 MB (2e2c3000)
    Free:                                273 MB (11102000)
    Total:                              1011 MB (3f3c5000)
Page File (Swap space):
    In Use:                             1321 MB (529a4000)
    Free:                               1105 MB (45169000)
    Total:                              2427 MB (97b0d000)
Virtual Memory (Address Space):
    In Use:                              887 MB (37723000)
    Free:                               1160 MB (488bd000)
    Total:                              2047 MB (7ffe0000)
Largest Contiguous Free Blocks:
     1. [at 4986b000]                    197 MB ( c585000)
     2. [at 3e1b9000]                    178 MB ( b2a7000)
     3. [at 1f5a0000]                    104 MB ( 6800000)
     4. [at 56032000]                     77 MB ( 4d3e000)
     5. [at 68b40000]                     70 MB ( 4660000)
     6. [at 3a320000]                     54 MB ( 3610000)
     7. [at 63568000]                     45 MB ( 2d48000)
     8. [at 35aff000]                     40 MB ( 2821000)
     9. [at 60f86000]                     37 MB ( 25ca000)
    10. [at 6f49d000]                     37 MB ( 25b3000)
                                        ======= ==========
                                         842 MB (34ac0000)

ans =

   207114240

You can't suppress the output, but it returns the largest memory block available ( 207,114,240 Bytes / 8 = 25,889,280 doubles )

Alex Kimmerly
  • 189
  • 1
  • 2
  • This command still works, but is not the same as the maximum possible array size, which should be checked with `m = memory; m.MaxPossibleArrayBytes`. Still, the error "Maximum variable size allowed by the program is exceeded" is not even related to this limit. It's related to a hard limit on the number of *elements* in an array, not a memory availability issue but an implementation limit. I've explained both concepts in detail [here](http://stackoverflow.com/a/28907084/2778484). – chappjc Mar 06 '15 at 20:38