0

I have this PHP code below to generate a set of 12 digit unique random numbers(ranging from 100000 to a million) and save it in db. I am first fetching the existing codes from MySQL db(right now there are already a million of them), flipping the array, generating a new codes. Later I use array_diff_key and array_keys on $random and $existingRandom to get the new codes which are to be saved back to db. 

// total labels is the number of new codes to generate
//$totalLabels is fetched as user input and could range from 100000 to a million
$codeObject = new Codes();
//fetch existing codes from db
$codesfromDB = $codeObject->getAllCodes();
$existingRandom = $random = array();
$existingRandom = $random = array_flip($codesfromDB);


$existingCount = count($random); //The codes you already have

do {
    $random[mt_rand(100000000000,999999999999)] = 1;
} while ((count($random)-$existingCount) < $totalLabels);
$newCodes = array_diff_key($random,$existingRandom);
$newCodes = array_keys($newCodes);

The issue I am facing is that the array_flip function is running out of memory and causing my program to crash Error 

"Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes)"

My questions are below:

1) Can someone help me understand why the array_flip is running out of memory. Memory limit in php.ini file is 256M. Please show me calculation of the memory used by the function if possible. (Also if array_flip passes array_diff_key and array_keys run out of memory)

2) How do I optimize the code so that the memory used is under the limit. I even tried to break the array_flip operation in smaller chunks but even that is running out of memory.

$size = 5000;
$array_chunk = array_chunk($codesfromDB, $size);

foreach($array_chunk as $values){
    $existingRandom[] = $random[] = array_flip($values);
}

3) Is what I am doing optimal would it be fair to further increase the memory limit in php.ini file. What are the things to keep in mind while doing that.

Here is my query as well to fetch the existing codes from db if needed:

    $sql = "SELECT codes FROM code";
    $stmt = $this->db->prepare($sql);
    $stmt->execute();
    $result = $stmt->fetchAll(PDO::FETCH_COLUMN, 0);
    return $result;
FBP
  • 345
  • 3
  • 15
  • How big is the inital array you fetch from the database? In KB, I mean? When you use the array-functions in PHP, PHP reads the array to be manipulated into memory, and if the array is too big, it will fail, always. So, it might be smart to process chunks, and by that I mean actually process chunks. Also, why don't you let the database handle the uniqueness of the keys? Make the column unique, and the database will not allow entries with equal keys. That way you can skip most of the array-handling in PHP. – junkfoodjunkie Nov 05 '16 at 08:03
  • @junkfoodjunkie how do I check the size from db? Also why I don't insert in SQL and let that handle is because I am inserting values all at once and if that fails I need to regenerate all values again. Given the number is large ~100000 to a million I will end in an endless loop doing that. Also I am not sure if there is any way to individually extract only the failed values and let the non duplicate values be inserted. – FBP Nov 05 '16 at 16:22
  • Of course there is - it's called a for-loop and PDO with parameterized queries, looping through the content of the generated array, catching errors and continuing the loop. And are you generating the values on the fly, or before looping and trying to insert them? If you do the latter, you will still have that array available, if you code it right. You could also in theory just do the generation while looping through the INSERTs, generating a new key until whatever quota is reached. That might work as well, and will probably be less taxing? – junkfoodjunkie Nov 05 '16 at 16:30
  • @junkfoodjunkie that I was doing earlier but using the for loop and inserting one value at a time is taking a lot of time as compared to inserting all values at once! – FBP Nov 07 '16 at 05:54
  • imo, it must be a MySQL routine like [this](http://stackoverflow.com/questions/11042546/how-do-i-populate-a-mysql-table-with-many-random-numbers). `fetchAll` is not the option. – Deadooshka Nov 07 '16 at 09:30

0 Answers0