Least memory intensive way to read a file in PHP

Question

I am reading a file containing around 50k lines using the file() function in Php. However, its giving a out of memory error since the contents of the file are stored in the memory as an array. Is there any other way?

Also, the lengths of the lines stored are variable.

Here's the code. Also the file is 700kB not mB.

private static function readScoreFile($scoreFile)
{
    $file = file($scoreFile);
    $relations = array();

    for($i = 1; $i < count($file); $i++)
    {
        $relation = explode("\t",trim($file[$i]));
        $relation = array(
                        'pwId_1' => $relation[0],
                        'pwId_2' => $relation[1],
                        'score' => $relation[2],
                        );
        if($relation['score'] > 0)
        {
            $relations[] = $relation;
        }
    }

    unset($file);
    return $relations;
}

Answer 1

Use fopen, fread and fclose to read a file sequentially:

$handle = fopen($filename, 'r');
if ($handle) {
    while (!feof($handle)) {
        echo fread($handle, 8192);
    }
    fclose($handle);
}

Answer 2

EDIT after update of question and comments to answer of fabjoa:

There is definitely something fishy if a 700kb file eats up 140MB of memory with that code you gave (you could unset $relation at the end of the each iteration though). Consider using a debugger to step through it to see what happens. You might also want to consider rewriting the code to use SplFileObject's CSV functions as well (or their procedural cousins)

SplFileObject::setCsvControl example

$file = new SplFileObject("data.csv");
$file->setFlags(SplFileObject::READ_CSV);
$file->setCsvControl('|');
foreach ($file as $row) {
    list ($fruit, $quantity) = $row;
    // Do something with values
}

---

For an OOP approach to iterate over the file, try SplFileObject:

SplFileObject::fgets example

$file = new SplFileObject("file.txt");
while (!$file->eof()) {
    echo $file->fgets();
}

SplFileObject::next example

// Read through file line by line
$file = new SplFileObject("misc.txt");
while (!$file->eof()) {
    echo $file->current();
    $file->next();
}

or even

foreach(new SplFileObject("misc.txt") as $line) {
    echo $line;
}

Pretty much related (if not duplicate):

• How to save memory when reading a file in Php?

Answer 3

If you don't know the maximum line length and you are not comfortable to use a magic number for the max line length then you'll need to do an initial scan of the file and determine the max line length.

Other than that the following code should help you out:

    // length is a large number or calculated from an initial file scan
    while (!feof($handle)) {
        $buffer = fgets($handle, $length);
        echo $buffer;
    }

Answer 4

Old question but since I haven't seen anyone mentioning it, PHP generators is a great way to reduce save memory consumption.

For example:

function read($fileName)
{
    $fileHandler = fopen($fileName, 'rb');

    while(($line = fgets($fileHandler)) !== false) {
        yield rtrim($line, "\r\n");
    }

    fclose($fileHandler);
}

foreach(read(__DIR__ . '/filenameHere') as $line) {
    echo $line;
}

Answer 5

allocate more memory during the operation, maybe something like ini_set('memory_limit', '16M');. Don't forget to go back to initial memory allocation once operation is done