Output streaming with PHP ob_start & ob_get_clean

Question

I have a script that echo out content in a php script and resulting in a very large file, e.g. 100MB

Currently I use the following way to capture the output and write to another file

ob_start();
require_once 'dynamic_data.php'; // echo 100MB data
$data = ob_get_clean();
file_put_contents($path, $data);

Are there any easy way to re-write the above program (better not touching dynamic_data.php as it is hard to re-factor) so it can stream the output to the file directly without saving the content in the memory?

Instead of echoing and recapturing the output can't you add the file operations directly in `dymanic_data.php` without echoing? Any constraints in doing this ? — shatheesh, Jun 08 '14 at 04:54
Because it is more easy to loop/echo using PHP as the template language instead of print to file directly. Also, I can reuse the same script for web output. — Ryan, Jun 08 '14 at 05:01

Martin Konecny · Answer 1 · 2014-06-08T05:28:16.077

The ob_start documentation provides a workaround for this. You need to pass in a $output_callback and a $chunk_size.

Say you set $chunk_size to 1MB. Then every 1MB of buffered output data, your $output_callback will be called with this data and you can flush it to disk (meanwhile the output buffer is implicitly flushed).

$output_callback = function($data) {
   //$buffer contains our 1MB of output

   file_put_contents($path, $data);

   //return new string buffer
   return "";
}

//call $output_callback every 1MB of output buffered.
ob_start($output_callback, 1048576);

require_once 'dynamic_data.php';

//call ob_clean at the end to get any remaining bytes 
//(implicitly calls $output_callback final time)
ob_clean();

score 0 · Answer 2 · answered Jun 08 '14 at 04:56

You can use proc_open and invoke the PHP interpreter with this file as argument. This will not store the data in memory but it'll create another process.

$descriptorspec = array(
   0 => array("pipe", "r"),  // stdin is a pipe that the child will read from
   1 => array("file", $path, "w"),  // stdout is a pipe that the child will write to
   2 => array("file", $path, "a") // stderr is a file to write to
);

$process = proc_open('php dynamic_data.php', $descriptorspec, $pipes);

if (is_resource($process)) {
    // $pipes now looks like this:
    // 0 => writeable handle connected to child stdin
    // 1 => readable handle connected to child stdout
    // Any error output will be appended to /tmp/error-output.txt

    fclose($pipes[0]);
    fclose($pipes[1]);
    $return_value = proc_close($process);
}
?>

Output streaming with PHP ob_start & ob_get_clean

2 Answers2