Memory usage increasing after every loop

Question

I have a foreach loop, reading a file containing about 200000 lines, this is my PHP :

foreach ($this->file as $row)
      {
        if ($this->file->valid())
        {

          //init client array
          $this->initInvoicesArray($row);
          $this->prepareInvoice();
          $row = null;
          $this->key = $this->file->key();

          //add msisdn to msisdn array, and client to clients array
          self::$MsisdnArray[] = $this->msisdn;
          self::$InvoicesArray[$this->msisdn] = $this->client;

          if ($i % 3000 == 0) 
          {
            //get valid users from table
            $this->prepareAndSaveValidClients();
          }

          $i++;
        }
      }

and this is the prepareAndSaveValidClients() function :

public function prepareAndSaveValidClients(){
    $query = Doctrine_Query::create()
        ->select('p.gender, p.email2, u.username, u.first_name, u.last_name, u.email_address, u.is_active, p.msisdn, p.user_id, p.city_id, p.street, p.zipcode, p.msisdn_status')
        ->from('sfGuardUser u')
        ->innerJoin('u.Profile as p ON u.id = p.user_id')
        ->whereIn('p.msisdn', self::$MsisdnArray)
        ->whereIn('p.status', self::$AllowedStatus);

    $results = $query->fetchArray();

    //instanciat an object collection for payment_notifications
    $collection = new Doctrine_Collection("payment_notifications");

    if (!empty($results))
    {
      foreach ($results as $key => $client)
      {
        $invoice = self::$InvoicesArray[$client['Profile']['msisdn']];

        $this->initInvoicesArray($invoice);
        $this->prepareInvoice();

        $this->prepareUserProfile($client);
        $this->prepareClient();

        $paymentNotifications = new paymentNotifications();
        $paymentNotifications->fromArray($this->client);

        $collection->add($paymentNotifications);

        $tel = $client['Profile']['msisdn'];
        $client = null;
      }

      $collection->save();

      //clear memory
      $results = null;
      $collection = null;
      self::$MsisdnArray = null;
      self::$InvoicesArray = null;


      $this->logSection('tel num', 'added :' . $tel . ' Memory usage : ' . memory_get_usage());
      $duration = microtime(true) - $this->startTime;
      $this->logSection('payment : ', sprintf('added in %s', $duration));
    }
  }

As for the functions :

   $this->initInvoicesArray($invoice);
   $this->prepareInvoice();

   $this->prepareUserProfile($client);
   $this->prepareClient();

They are just for preparing $this->client

This is Memory Usage as it's displayed for every loop :

>> tel num   added :0699946185 Memory usage : 89287596
>> payment :  added in 8.6373870372772
>> tel num   added :0699983919 Memory usage : 165854544
>> payment :  added in 18.373502969742
>> tel num   added :0699949623 Memory usage : 241338788
>> payment :  added in 29.336947917938
>> tel num   added :0699854750 Memory usage : 319173092
>> payment :  added in 40.880628824234

As you can see, I tried to free memory of these variables :

$results = null;
$collection = null;
self::$MsisdnArray = null;
self::$InvoicesArray = null;

But in vain, Memory Usage keeps increasing after every loop, which results in a An Allowed Memory size exausted fatal error. How can I optimize it?

Thank you.

Which PHP version are you using? The Garbage Collector got a major update from 5.2 to 5.3 which exactly solved your problem. — Marc, Apr 14 '15 at 08:36
@SanjayKumarNS, now I close the file after reading, but the same problem persists — SmootQ, Apr 14 '15 at 08:53
Read Mode : `$this->file = new SplFileObject($this->filePath, 'r');` `$this->file->setFlags(SplFileObject::READ_CSV);` — SmootQ, Apr 14 '15 at 09:28
@Marc, Sorry I didn't notice your comment , I'm using 5.3, and yes, this is what I've learned from this page http://php.net/manual/en/features.gc.performance-considerations.php .. thank you for your comment — SmootQ, Apr 14 '15 at 14:18
Okay, I would anyway recommend you to read http://php.net/manual/en/features.gc.performance-considerations.php if it won't help it is at least good to know — Marc, Apr 14 '15 at 14:20

score 3 · Answer 1 · answered Apr 14 '15 at 09:07

3

Doctrine objects have circular references that php has trouble to free. To solve this, doctrine provides free() method, call it on the collection just before setting it to null:

$collection->free(true);
$collection = null;

answered Apr 14 '15 at 09:07

Marek

7,337
1
22
33

1

Bravo my dear friend +1, I added $collection->free() and this is current result Memory Usage: 59681588, 106647676, 152936232...etc .. now it stops after 10 loops (before it crashes after 7 loops). Not bad , thank you so much. But the Memory Usage is still increasing. – SmootQ Apr 14 '15 at 09:21
1

Doctrine_Query has also `free()` method, try that. – Marek Apr 14 '15 at 09:35
I will add it too, and see – SmootQ Apr 14 '15 at 09:39
It still runs 10 loops then sends the Memory fatal error, but the Memory usage is reduced again at every loop, thank you so much again. – SmootQ Apr 14 '15 at 09:45
1

I'm afraid your only option is to execute separate process for each X lines in the file. – Marek Apr 14 '15 at 11:18
this is what I tought about, I will suggest to create a batch that executes like 10 processes. Thank you so much for your help – SmootQ Apr 14 '15 at 12:21
Forget to tell you that I solved the problem by using mere pdo and mysql (without using doctrine) for the select query.... the Memory usage remained stable... It's been about 3 months, just for the record. – SmootQ Jun 25 '15 at 21:02

score 1 · Answer 2 · answered May 27 '15 at 13:31

1

After calling $this->prepareAndSaveValidClients(); you could also use the gc_collect_cycles() function. It will help with possible memory leaks.

answered May 27 '15 at 13:31

lsouza

2,448
4
26
39

Thank you , I solved the problem by using sql and pdo directly, (without doctrine) in the select query. – SmootQ Jun 14 '15 at 10:23

score 0 · Accepted Answer · answered Jun 25 '15 at 21:04

0

I solved the problem 2 months ago, and the Memory_usage became stable, but I forgot to post the solution here.

What I did, is that I removed Doctrine, and I used mere PDO and Sql for the select query.

answered Jun 25 '15 at 21:04

SmootQ

2,096
7
33
58

Memory usage increasing after every loop

3 Answers3