How to read a range of rows from CSV file to JSON array using PHP to handle large CSV file?

Question

The target is how to read a range of rows/lines from large CSV file into a JSON array in order to handle large files and read the data in pagination method, each page fetches a range of lines ( e.x. page number 1 fetch from line 1 to 10, page number 2 fetch from line 11 to line 20, and so and ).

the below PHP script read from the being CSV file to the desired line ($desired_line), My question is how we can determine the starting line to read from a specific line ($starting_line)

<?php
// php function to convert csv to json format
function csvToJson($fname, $starting_line, $desired_line) {
    // open csv file
    if (!($fp = fopen($fname, 'r'))) {
        die("Can't open file...");
    }
    
    //read csv headers
    $key = fgetcsv($fp,"1024","\t");
    
    $line_counter = 0; 
 
    
    // parse csv rows into array
    $json = array();
        while (($row = fgetcsv($fp,"1024","\t")) && ($line_counter < $desired_line)) {
        $json[] = array_combine($key, $row);
        $line_counter++; 
    }
    
    // release file handle
    fclose($fp);
    
    // encode array to json
    return json_encode($json);
}



// Define the path to CSV file
$csv = 'file.csv';
print_r(csvToJson($csv, 20, 30));

?>

Well if the condition `($line_counter < $desired_line)` applies an _upper_ limit, then what the counterpart for that would be, to apply a lower limit "in the other direction", should be pretty obvious, is it not? — CBroe, Jan 14 '22 at 13:54
Incorporate PHP Generator to generate line by line directly to JSON string: [PHP - json\_encode a generator object (using yield)](https://stackoverflow.com/questions/36446268/php-json-encode-a-generator-object-using-yield) — Justinas, Jan 14 '22 at 13:55
You have `$starting_line` and `$desired_line` there. How about you start with figuring out how to pass them as arguments to your function; then tweak your condition. And then do a bit of research as to _whether it's possible to read specific lines only_ instead of looping until you get there. Imagine the wasted cycles with that approach in `start 10000, until 10100`... — Markus AO, Jan 14 '22 at 14:30
@MarkusAO >> For pass the starting_line and desired_line as arguments, I updated the function — Sam., Jan 14 '22 at 17:16

Flash Thunder · Accepted Answer · 2022-01-14T17:18:24.433

You should use functions like:

fgets() to read the file line by line

fseek() to move to the position of the last fgets() of the chunk

ftell() to read the position for fseek()

Something like this (it's only a schema):

<?php

...

$line_counter = 0;
$last_pos = ...
$fseek($fp,$last_pos);
while($line = fgets($fp)){ // read a line of the file
   $line_counter++;
   (...) // parse line of csv here
   if($line_counter == 100){
      $lastpos = ftell($fp);
      (...) // save the $lastpos for next reading cycle
      break;
   }
}

...

?>

You can also skip the fseek() and ftell() part and just count the lines every time from the beginning, but that will generally have to go through the whole file from the beginning till the desired lines.

How to read a range of rows from CSV file to JSON array using PHP to handle large CSV file?

1 Answers1