Fastest way to search and remove/add line from/to file

Question

I'm wondering what the fastest way is to remove a specific line from a potentially large file, if it exists.

So for example I want the following line

abc

to be removed from the file

yxz
srtabc
abc
efg

becoming

yxz
srtabc
efg

Also, I want to add a line to the file, if it doesn't already exist.

The order of the file doesn't matter, and can be sorted if it provide a performance boost to the search.

Performance has never been my strong point, so I'm having a little trouble determining the best path to go down here.

Does the solution have to be programatically ? – Nir Alfasi Oct 10 '12 at 23:02 — Nir Alfasi, Oct 10 '12 at 23:02

Baba · Answer 1 · 2012-10-10T23:28:26.947

I don't advice using file_put_contents or file_get_contents because it would load all the contents of the file into PHP at once and this would not work for you if you are working with large file .

You can work with 2 files .. and read them a line after the other ..... replace anything you need to replace then rename at the end .... This is only efficient when working with very large file

Prof Of Concept

set_time_limit(0);

$baseFile = "log.txt";
$tempFile = $baseFile . ".temp";

touch($tempFile);

$findAndReplace = array("abc" => "","efg"=>"WWW");

$fileTemp = fopen($tempFile, "a+");
$fileBase = fopen($baseFile,"r");

while ( !feof($fileBase))  {
    $var = trim(fgets($fileBase));
    if (array_key_exists($var, $findAndReplace)) {
        var_dump($var);
        $var = $findAndReplace[$var];
    }
    if (!empty($var))
        fwrite($fileTemp , $var . PHP_EOL);
}

fclose($fileBase);
fclose($fileTemp);
unlink($baseFile);
rename($tempFile, $baseFile);

score 0 · Answer 2 · answered Oct 10 '12 at 23:06

For performance, don't use php.

But if you insist, and if all you need is a single replace operation, read in the entire file, then replace the line (as a string), and write it back. I.e. don't split the file into separate lines, which would make things slower than necessary.

This should to the trick:

$input = file_get_contents($filename_in);

$find = "abc..";

$find_q = preg_quote($find,'/');
$output = preg_replace("/^$find_q(\n|\$)/m","",$input);

file_put_contents($filename_out,$output);

It removes the lines containing exactly $find, including the trailing newline. In case the last line is not \n terminated, it will still be removed.

To test if a line is present, take a similar approach:

$find_q = preg_quote($find,'/');
if ( !preg_match("/^$find_q(\n|\$)/m",$input) )
{
    $input .= $find."\n"; // note: this assumes that $input is \n terminated
}

Fastest way to search and remove/add line from/to file

2 Answers2