9

So I've been trying my hands on laravel's chunking in Eloquent but I've run into a problem. Consider the following code (a much more simplified version of my problem):

$data = DB::connection('mydb')->table('bigdata')
->chunk(200, function($data) {
  echo memory_get_usage();
  foreach($data as $d) {
    Model::create(
      array(
        'foo' => $d->bar,
        ...
        //etc
      ));
  }
}

So when I run the following code my memory outputs look like this:

19039816
21490096
23898816
26267640
28670432
31038840

So without jumping into php.ini and changing the memory_limit value any clue why it isn't working? According to the documentation: "If you need to process a lot (thousands) of Eloquent records, using the chunk command will allow you to do without eating all of your RAM".

I tried unset($data) after the foreach function but it did not help. Any clue as to how I can make use of chunk or did I misinterpret what it does?

tiffanyhwang
  • 1,413
  • 4
  • 18
  • 26
  • 4
    Okay with a little digging around adding `DB::disableQueryLog()` before our query will solve our problem. – tiffanyhwang Jan 14 '14 at 11:37
  • 1
    As you found out yourself, the query logging is eating away the memory. With that said, you could also adjust the memory limit for a single class in the constructor, like so: `ini_set('memory_limit', $this->memorylimit);` – nielsstampe Jan 14 '14 at 12:14
  • 1
    Use the Laravel debugbar to get a better insight –  Jun 08 '16 at 09:43

2 Answers2

1

Chunking data doesn't reduce memory usage, you need to do it like pagination directly using the database.

Like first fetch starting 200 order by id or something, and after processing first 200, fire that query again with a where clause asking next 200 results.

Rohit Khatri
  • 1,980
  • 3
  • 25
  • 45
1

You can use lazy collections to improve memory uses for a big collection of data. It uses PHP generators under the hood. Take a look at the cursor example here https://laravel.com/docs/5.4/eloquent#chunking-results

Kamlesh Suthar
  • 182
  • 2
  • 8