1

I'm using The PHP League CSV importer/exporter to import a large CSV file in Laravel. Since the file is large, I would like to stream it to the CSV parser and handle it one line at a time, without loading every line into memory.

Laravel uses flysystem for the underlying filesystem, and I am using that to obtain a PHP resource to the source CSV.

What I don't understand is how - if it is at all possible - I can feed that resource stream into League CSV so that it reads one line at a time for me to process, before reading in the next line. All the documentation seems to imply that a CSV file is always read fully into memory, and that is what I want to avoid.

Do I need to use callbacks? If so, how can I be sure the stream resource is only being read one line at a time as needed, and not all at once?

I'm guessing I start by creating a stream reader?

use League\Csv\Reader;
$reader = Reader::createFromStream($resource, 'r');
Jason
  • 4,411
  • 7
  • 40
  • 53

1 Answers1

1

You can iterate over the rows without loading the whole file by using the IteratorAggregate interface of the Reader. So you basically just do

foreach ($reader as $row) {
    // do stuff
}

If you are using a mac to read or create the CSV file you will need to add this to your code for it to work correctly:

if (!ini_get("auto_detect_line_endings")) {
    ini_set("auto_detect_line_endings", '1');
}
VaaChar
  • 688
  • 8
  • 20
  • Thank you. Having run some memory tests, it looks like the CSV package reverts to resources internally anyway, even if given a local pathname rather than a stream or resource. It kind of *does the right thing* with the `IteratorAggregate` no matter what you throw at it. After `foreach` looping over 40k long CSV rows, the memory usage increases by a few 100kbytes, which is exactly what I needed to see. – Jason Oct 16 '18 at 12:19