3

How do I pre-allocate memory for an array in PHP? I want to pre-allocate space for 351k longs. The function works when I don't use the array, but if I try to save long values in the array, then it fails. If I try a simple test loop to fill up 351k values with a range(), it works. I suspect that the array is causing memory fragmentation and then running out of memory.

In Java, I can use ArrayList al = new ArrayList(351000);.

I saw array_fill and array_pad but those initialize the array to specific values.

Solution:

I used a combination of answers. Kevin's answer worked alone, but I was hoping to prevent problems in the future too as the size grows.

ini_set('memory_limit','512M');
$foundAdIds = new \SplFixedArray(100000); # google doesn't return deleted ads. must keep track and assume everything else was deleted.
$foundAdIdsIndex = 0;
// $foundAdIds = array();
$result = $gaw->getAds(function ($googleAd) use ($adTemplates, &$foundAdIds, &$foundAdIdsIndex) { // use call back to avoid saving in memory
  if ($foundAdIdsIndex >= $foundAdIds->count()) $foundAdIds->setSize( $foundAdIds->count() * 1.10 ); // grow the array
  $foundAdIds[$foundAdIdsIndex++] = $googleAd->ad->id; # save ids to know which to not set deleted
  // $foundAdIds[] = $googleAd->ad->id;
Community
  • 1
  • 1
Chloe
  • 25,162
  • 40
  • 190
  • 357
  • 1
    What do you mean by "when I don't use the array" and "then it fails"? What happens? Do you see any error messages? – gen_Eric Sep 09 '15 at 13:53
  • 1
    Is there a reason you don't want to initialize the array elements to a specific value? – Jeff Lambert Sep 09 '15 at 13:54
  • possible duplicate of [Is there a PHP equivalent of "new Array (number)" in javascript?](http://stackoverflow.com/questions/11404398/is-there-a-php-equivalent-of-new-array-number-in-javascript) – Benjamin Sep 09 '15 at 13:57
  • take a look to JudyArray : http://php.net/manual/en/book.judy.php or SPL Datastructures: http://php.net/manual/en/spl.datastructures.php – Halayem Anis Sep 09 '15 at 13:58
  • 1
    "The function works when I don't use the array" - can we see the code of the function? – user4035 Sep 09 '15 at 13:58
  • 1
    What problem are you trying to solve by pre-allocating memory? If you are running out of memory with a Fatal Error then the solution presented below by Kevin is the answer to your question. Determine how much memory you need to store an array of `N` longs, add on any additional memory you may need and then set the `memory_limit` for the script to something that is large enough. – Josh J Sep 09 '15 at 14:02

4 Answers4

5

PHP has an Array Class with SplFixedArray

$array = new SplFixedArray(3);
$array[1] = 'test1';
$array[0] = 'test2';
$array[2] = 'test3';
foreach ($array as $k => $v) {
    echo "$k => $v\n";
}
$array[] = 'fails';

gives

0 => test1

1 => test2

2 => test3

Community
  • 1
  • 1
Syed mohamed aladeen
  • 6,507
  • 4
  • 32
  • 59
  • Unfortunately, I tried this and tried to grow the array by 10% each time it reaches capacity and it still failed with an out of memory error (elsewhere, not during resize). `if ($foundAdIdsIndex >= $foundAdIds->count()) $foundAdIds->setSize( $foundAdIds->getSize() * 1.10 );` – Chloe Sep 09 '15 at 21:55
2

The quick answer is: you can't

PHP is quite different from java.

You can make an array with specific values as you said, but you already know about them. You can 'fake' it by filling it with null values, but that's about the same to be honest.

So unless you want to just create one with array_fill and null (which is a hack in my head), you just can't.

(You might want to check your reasoning about the memory. Are you sure this isn't an XY-problem? As memory is limited by a number (max usage) I don't think the fragmentation would have much effect. Check what is taking your memory rather then try going down this road)

Nanne
  • 64,065
  • 16
  • 119
  • 163
  • Yes I'm sure, because I've already made the code quite complicated with callbacks rather than returning blocks of objects. When I comment out the array of ids and don't fill it, there is no memory problem. Yet in a separate test I can fill an array of that size without problem. It is the combination of filling the integer array and the other code that is causing the problem, but the other code is in a function that is called in blocks of 2000 and goes out of scope, freeing up any memory it might use. – Chloe Sep 09 '15 at 22:04
  • Ok I didn't know it at the time, but this turned into an XY problem. It turns out MySQL doesn't like a 300k+ item `NOT IN` clause and practically locks up. So I researched the web service a bit more and found I can return deleted items by making two separate calls, one specifically asking for deleted items. – Chloe Sep 10 '15 at 00:15
2

As other people have pointed out, you can't do this in PHP (well, you can create an array of fixed length, but that's not really want you need). What you can do however is increase the amount of memory for the process.

ini_set('memory_limit', '1024M');

Put that at the top of your PHP script and you should be ok. You can also set this in the php.ini file. This does not allocate 1GB of memory to PHP, but rather allows PHP to expand it's memory usage up to that point.

A couple of things to point out though:

  • This might not be allowed on some shared hosts
  • If you're using this much memory, you might need to have a look at how you're doing things and see if they can be done more efficiently
  • Look out for opportunities to clear out unneeded resources (do you really need to keep hold of $x that contains a huge object you've already used?) using unset($x);
Kevin Nagurski
  • 1,889
  • 11
  • 24
  • Heroku allows `ini_set`, but their dyno (VPS) limit is 512M. I will just increase the `memory_limit`. I already use a callback function to avoid returning a list of ActiveRecord/DBO objects, and work on them one at a time. However, the web service I'm using doesn't return deleted items, so I have to keep track of which ARE returned and mark the REST as deleted. – Chloe Sep 09 '15 at 21:19
1

The closest you will get is using SplFixedArray. It doesn't preallocate the memory needed to store the values (because you can't pre-specify the type of values used), but it preallocates the array slots and doesn't need to resize the array itself as you add values.

deceze
  • 510,633
  • 85
  • 743
  • 889
  • 1
    But it still wouldn't do anything for the memory problem is my guess? – Nanne Sep 09 '15 at 13:58
  • 1
    Hard to say. It's unclear what the problem is in the first place. Perhaps rejiggling the memory for the array expansion is the real problem, and using an SplFixedArray is just slightly more efficient enough to avoid that problem. OP may be dancing close to the limit there, so this *could* make a difference. – deceze Sep 09 '15 at 14:00
  • 1
    According to the docs, `SplFixedArray` provides faster array access. It does not explicitly state that it is more memory efficient (although it may be). It also will not increase the amount of memory a php script can consume which seems to be OPs problem even if OP does not realize it yet. – Josh J Sep 09 '15 at 14:05
  • I freely admit that I'm speculating here and that OP's problem may be something entirely else. But if you know how many elements you'll have, SplFixedArray is worth considering either way... – deceze Sep 09 '15 at 14:11
  • Unfortunately, it still runs out of memory. No, I cannot know how many elements there will be in total, just a rough guess. I tried to resize by 10% dynamically but it failed elsewhere and ran out of memory. `if ($foundAdIdsIndex >= $foundAdIds->count()) $foundAdIds->setSize( $foundAdIds->count() * 1.10 );` – Chloe Sep 09 '15 at 22:10