0

I am importing a CSV using maatwebsite/excel package. The CSV contains over 100 million rows, so I'm using WithChunkReading. I'm also logging each row insertion.

protected $rowCount = 0;

public function collection(Collection $rows): bool
{
    foreach ($rows as $row)
    {
        // Insertions here

        ++$this->rowCount;
        Log::debug('Inserted row #'.$this->rowCount);
    }
    return false;
}

public function chunkSize(): int
{
    return 10;
}

The problem is that after every chunk, $rowCount resets to zero.

This is what laravel.log outputs:

[2020-07-13 10:06:10] local.DEBUG: Inserted row #1
[2020-07-13 10:06:11] local.DEBUG: Inserted row #2
[2020-07-13 10:06:11] local.DEBUG: Inserted row #3
[2020-07-13 10:06:11] local.DEBUG: Inserted row #4
[2020-07-13 10:06:11] local.DEBUG: Inserted row #5
[2020-07-13 10:06:12] local.DEBUG: Inserted row #6
[2020-07-13 10:06:12] local.DEBUG: Inserted row #7
[2020-07-13 10:06:12] local.DEBUG: Inserted row #8
[2020-07-13 10:06:12] local.DEBUG: Inserted row #9
[2020-07-13 10:06:13] local.DEBUG: Inserted row #10
[2020-07-13 10:06:16] local.DEBUG: Inserted row #1
[2020-07-13 10:06:16] local.DEBUG: Inserted row #2
[2020-07-13 10:06:16] local.DEBUG: Inserted row #3

I also tried using $rowCount by reference in a closure

public function collection(Collection $rows): bool
{
    $rowCount = 0;
    foreach ($rows as $row)
    {
        // Insertions here

        $counter = function() use(&$rowCount){
            return ++$rowCount;
        };
        Log::debug('Inserted row #'.$counter->call($row));
    }
    return false;
}

But the result is still the same.

Update: I tried accessing the counter from outside the collection method -

protected $rowCount = 0;

public function collection(Collection $rows): bool
{
    foreach ($rows as $row)
    {
        // Insertions here

        $rowCount = $this->incrementCounter();
        Log::debug('Inserted row #'.$rowCount);
    }
    return false;
}

protected function incrementCounter(){
    return ++$this->rowCount;
}

public function chunkSize(): int
{
    return 10;
}

This, however, did not help either. The counter still resets back.

How can I persist the value of $rowCount so that it does not reset back to 0 after every chunk?
TIA!

Qumber
  • 13,130
  • 4
  • 18
  • 33
  • Have you tried the `RemembersRowNumber` trait mentioned [here](https://docs.laravel-excel.com/3.1/imports/chunk-reading.html#chunk-reading)? – Remul Jul 13 '20 at 11:16
  • @Remul, "Remembering row numbers is only intended for `ToModel` imports." Unfortunately, I can't use them because I need to update the record if it already exists - using `updateOrCreate`. I can either use `ToCollection` or `OnEachRow`, neither of these work with `RemembersRowNumber`. :( – Qumber Jul 13 '20 at 11:43
  • As far as I know `toModel` will just call `saveOrFail()` on the provided model, so you can either return a new model instance or a existing one which then would be updated. – Remul Jul 13 '20 at 11:58
  • So you could change your code to use `ToModel`, then use `YourModel::firstOrNew` to either retrieve the existing model or create a new model instance, fill all the attributes that you want to update and then return the model, which would either create a new record or update existing ones. – Remul Jul 13 '20 at 12:00
  • @Remul, It turns out that the trait is not available in the package yet. See the [issue I created yesterday](https://github.com/Maatwebsite/Laravel-Excel/issues/2755). I tried creating the trait manually, getting the source from their repository, but it was causing all sorts of issues - returning `int` instead of `resource` to the model function, etc. Additionally, I wouldn't want my client to have to add files manually to the vendor directory. – Qumber Jul 14 '20 at 06:05

0 Answers0