Laravel Large CSV import using maatwebsite excel

Question

I have a CSV file with 30 k rows. I have used maatwebsite excel to import CSV in my pgsql database.

Problem is every time it's uploading 10k-12k data in database, then page giving HTTP ERROR 500

Error :

This page isn’t working
localhost is currently unable to handle this request.
HTTP ERROR 500

I have changes below variable in php.ini

max_execution_time=0
max_input_time=3000
post_max_size=128M

I have tried below code in ReportsImport

class UserReport implements ToModel, WithCustomCsvSettings,WithChunkReading,WithHeadingRow
{
    public function model(array $row)
    {
          // dd($row);
          return new UserReport([
              'user'     => $row['username],
              'amount'   => $row['amount']
          ]);
    }
    
    public function getCsvSettings(): array
    {
         return [
              'input_encoding' => 'UTF-8'
         ];
    }
    
    public function chunkSize(): int
    {
        return 1000;
    }
}

How can I resolved this HTTP 500 error ?

Error Log : local.ERROR: Allowed memory size of 536870912 bytes exhausted

Version : Laravel Framework 7.26.1

@Thomas local.ERROR: Allowed memory size of 536870912 bytes exhausted — Niloy Rony, Sep 01 '20 at 16:34
Did you add the `implements WithChunkReading`? What happens if you decrease the cunk size? — Thomas, Sep 01 '20 at 16:37
@Thomas I have updated my code, shared full class what I have written. I have reduced chunk size 1000 to 200 only 800 increased ! — Niloy Rony, Sep 01 '20 at 16:48
Did you try decreasing the chunk size? If that doesn't help I would try adding [Batch inserts](https://docs.laravel-excel.com/3.1/imports/chunk-reading.html#using-it-together-with-batch-inserts) next (in addition to the existing Chunk reading). — Thomas, Sep 01 '20 at 16:52
@Thomas After use Batch Insert It has worked !! Thank you so much ! Would you please add this as an answer ? — Niloy Rony, Sep 01 '20 at 16:59

score 2 · Accepted Answer · answered Sep 01 '20 at 17:06

The problem is that the application is trying to hold too much data in memory. I see you're already using Chunk reading but it appears not to be enough

To decrease the memory consumption when reading the table data, try decreasing your chunk size.

To decrease the memory consumption of the models, try adding Batch inserts in addition to chunk reading .

Laravel Large CSV import using maatwebsite excel

1 Answers1