Redirect pipe acquired by proc_open() to file for remainder of process duration

Question

Say, in PHP, I have bunch of unit tests. Say they require some service to be running.

Ideally I want my bootstrap script to:

start up this service
wait for the service to attain a desired state
hand control to the unit-testing framework of choice to run the tests
clean up when the tests end, gracefully terminating the service as appropriate
set up some way of capturing all output from the service along the way for logging and debugging

I'm currently using proc_open() to initialize my service, capturing the output using the pipe mechanism, checking that the service is getting to the state I need by examining the output.

However at this point I'm stumped - how can I capture the rest of the output (including STDERR) for the rest of the duration of the script, while still allowing my unit tests to run?

I can think of a few potentially long-winded solutions, but before investing the time in investigating them, I would like to know if anyone else has come up against this problem and what solutions they found, if any, without influencing the response.

Edit:

Here is a cutdown version of the class I am initializing in my bootstrap script (with new ServiceRunner), for reference:

<?php


namespace Tests;


class ServiceRunner
{
    /**
     * @var resource[]
     */
    private $servicePipes;

    /**
     * @var resource
     */
    private $serviceProc;

    /**
     * @var resource
     */
    private $temp;

    public function __construct()
    {
        // Open my log output buffer
        $this->temp = fopen('php://temp', 'r+');

        fputs(STDERR,"Launching Service.\n");
        $this->serviceProc      = proc_open('/path/to/service', [
            0 => array("pipe", "r"),
            1 => array("pipe", "w"),
            2 => array("pipe", "w"),
        ], $this->servicePipes);

        // Set the streams to non-blocking, so stream_select() works
        stream_set_blocking($this->servicePipes[1], false);
        stream_set_blocking($this->servicePipes[2], false);

        // Set up array of pipes to select on
        $readables = [$this->servicePipes[1], $this->servicePipes[2]);

        while(false !== ($streams = stream_select($read = $readables, $w = [], $e = [], 1))) {
            // Iterate over pipes that can be read from
            foreach($read as $stream) {
                // Fetch a line of input, and append to my output buffer
                if($line = stream_get_line($stream, 8192, "\n")) {
                    fputs($this->temp, $line."\n");
                }

                // Break out of both loops if the service has attained the desired state
                if(strstr($line, 'The Service is Listening' ) !== false) {
                    break 2;
                }

                // If the service has closed one of its output pipes, remove them from those we're selecting on
                if($line === false && feof($stream)) {
                    $readables = array_diff($readables, [$stream]);
                }
            }
        }

        /* SOLUTION REQUIRED SOLUTION REQUIRED SOLUTION REQUIRED SOLUTION REQUIRED */
        /* Set up the pipes to be redirected to $this->temp here */

        register_shutdown_function([$this, 'shutDown']);
    }

    public function shutDown()
    {
        fputs(STDERR,"Closing...\n");
        fclose($this->servicePipes[0]);
        proc_terminate($this->serviceProc, SIGINT);
        fclose($this->servicePipes[1]);
        fclose($this->servicePipes[2]);
        proc_close($this->serviceProc);
        fputs(STDERR,"Closed service\n");

        $logFile = fopen('log.txt', 'w');

        rewind($this->temp);
        stream_copy_to_stream($this->temp, $logFile);

        fclose($this->temp);
        fclose($logFile);
    }
}

well as I understand it, you are testing a unit (X) that depends on a service (Y) and you want the framework to bootstrap it and hand it to the unittest framework so the unittest of (X) can use the service it self. If what I described is what you need, then you are doing it wrong, actually, this is a well know issue when you do unittest, and they have a well known solution for it, it is called mocking (doubles). You have to ask your unittest framework to create a mock of the service instead of asking your bootstrap to initialize the real service. If you put some codes I can be more helpful :) — Ahmad Hajjar, May 12 '17 at 03:41
I agree that this is the case when the service is required by the system under test, and I would be taking that approach if it was the case. However the service I am running is required by the code I am using to test the system, (think like a network-based test case provider) so a mock doesn't help me here. — Benjamin, May 12 '17 at 05:37
So you want to run the service as a process in the code and you want to grab the STDERR ? if that's the case `open_proc` provides a set of descriptors you can use them to define IO http://php.net/manual/en/function.proc-open.php#example-4226 — Ahmad Hajjar, May 12 '17 at 06:09
Yes. I am already doing this, the issue I have is that having acquired some pipes representing STDERR and STDOUT, and then waiting on them until a certain line of output is reached (startup of the service is not instantaneous), how do I then capture the rest of the output from both output pipes in the background, while still allowing the unit tests to run. — Benjamin, May 12 '17 at 06:23
ok this is interesting :) so you need the output asynchronously? You just need it ! but without having the unittest to wait until the end of the service execution hmmmmmm do you have any control over the code of the service you are depending on? — Ahmad Hajjar, May 12 '17 at 06:34
Not particularly easily, no. Take it that the solution has to be in the PHP bootstrap script itself, in the way the pipes and process(es) are manipulated.I'm pretty sure this isn't a Kobayashi Maru problem :) — Benjamin, May 12 '17 at 07:30
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/144037/discussion-between-benjamin-and-ahmad-hajjar). — Benjamin, May 12 '17 at 08:22

score 1 · Answer 1 · answered May 14 '17 at 16:14

Suppose the service is implemented as service.sh shell script with the following contents:

#!/bin/bash -
for i in {1..4} ; do
  printf 'Step %d\n' $i
  printf 'Step Error %d\n' $i >&2
  sleep 0.7
done

printf '%s\n' 'The service is listening'

for i in {1..4} ; do
  printf 'Output %d\n' $i
  printf 'Output Error %d\n' $i >&2
  sleep 0.2
done

echo 'Done'

The script emulates startup process, prints the message indicating that the service is ready, and prints some output after startup.

Since you are not proceeding with the unit tests until the "service-ready marker" is read, I see no special reason to do this asynchronously. If you want to run some process (updating UI etc.) while waiting for the service, I would suggest using an extension featuring asynchronous functions (pthreads, ev, event etc.).

However, if there are only two things to be done asynchronously, then why not fork a process? The service can run in the parent process, and the unit tests can be launched in the child process:

<?php
$cmd = './service.sh';
$desc = [
  1 => [ 'pipe', 'w' ],
  2 => [ 'pipe', 'w' ],
];
$proc = proc_open($cmd, $desc, $pipes);
if (!is_resource($proc)) {
  die("Failed to open process for command $cmd");
}

$service_ready_marker = 'The service is listening';
$got_service_ready_marker = false;

// Wait until service is ready
for (;;) {
  $output_line = stream_get_line($pipes[1], PHP_INT_MAX, PHP_EOL);
  echo "Read line: $output_line\n";
  if ($output_line === false) {
    break;
  }
  if ($output_line == $service_ready_marker) {
    $got_service_ready_marker = true;
    break;
  }

  if ($error_line = stream_get_line($pipes[2], PHP_INT_MAX, PHP_EOL)) {
    $startup_errors []= $error_line;
  }
}

if (!empty($startup_errors)) {
  fprintf(STDERR, "Startup Errors: <<<\n%s\n>>>\n", implode(PHP_EOL, $startup_errors));
}

if ($got_service_ready_marker) {
  echo "Got service ready marker\n";
  $pid = pcntl_fork();
  if ($pid == -1) {
    fprintf(STDERR, "failed to fork a process\n");

    fclose($pipes[1]);
    fclose($pipes[2]);
    proc_close($proc);
  } elseif ($pid) {
    // parent process

    // capture the output from the service
    $output = stream_get_contents($pipes[1]);
    $errors = stream_get_contents($pipes[2]);

    fclose($pipes[1]);
    fclose($pipes[2]);
    proc_close($proc);

    // Use the captured output
    if ($output) {
      file_put_contents('/tmp/service.output', $output);
    }
    if ($errors) {
      file_put_contents('/tmp/service.errors', $errors);
    }

    echo "Parent: waiting for child processes to finish...\n";
    pcntl_wait($status);
    echo "Parent: done\n";
  } else {
    // child process

    // Cleanup
    fclose($pipes[1]);
    fclose($pipes[2]);
    proc_close($proc);

    // Run unit tests
    echo "Child: running unit tests...\n";
    usleep(5e6);
    echo "Child: done\n";
  }
}

Sample Output

Read line: Step 1
Read line: Step 2
Read line: Step 3
Read line: Step 4
Read line: The service is listening
Startup Errors: <<<
Step Error 1
Step Error 2
Step Error 3
Step Error 4
>>>
Got service ready marker
Child: running unit tests...
Parent: waiting for child processes to finish...
Child: done
Parent: done

This wasn't what I ended up doing, and not something I had available to me, as my target platform didn't have the process control extension enabled, or available. However this may be useful to some people, so I have awarded the bounty here. I'll post my answer shortly. — Benjamin, May 15 '17 at 09:08

score 0 · Answer 2 · answered May 15 '17 at 07:26

0

You can use the pcntl_fork() command to fork the current process to do both tasks and wait for the tests to finish:

 <?php
 // [launch service here]
 $pid = pcntl_fork();
 if ($pid == -1) {
      die('error');
 } else if ($pid) {
      // [read output here]
      // then wait for the unit tests to end (see below)
      pcntl_wait($status);
      // [gracefully finishing service]
 } else {
      // [unit tests here]
 }

 ?>

answered May 15 '17 at 07:26

Adam

17,838
32
54

1

Isn't this exactly what Ruslan Osmanov is proposing in the previous answer? – Benjamin May 15 '17 at 08:08
1

@Benjamin You are right, both use the `pcntl_fork()` function. I have to admit, that I didn't read that part of the answer which was quite lost in his post. His answer is very comprehensive to the current problem, but I will let my answer for people wanting generic clues. – Adam May 15 '17 at 08:36
(and give him the bounty) – Adam May 15 '17 at 08:40

score 0 · Accepted Answer · edited May 23 '17 at 12:18

What I ended up doing, having reached the point where the service had been initialized correctly, was to redirect the pipes from the already opened process as the standard input to a cat process per-pipe, also opened by proc_open() (helped by this answer).

This wasn't the whole story, as I got to this point and realised that the async process was hanging after a while due to the stream buffer filling up.

The key part that I needed (having set the streams to non-blocking previously) was to revert the streams to blocking mode, so that the buffer would drain into the receiving cat processes correctly.

To complete the code from my question:

// Iterate over the streams that are stil open
foreach(array_reverse($readables) as $stream) {
    // Revert the blocking mode
    stream_set_blocking($stream, true);
    $cmd = 'cat';

    // Receive input from an output stream for the previous process,
    // Send output into the internal unified output buffer
    $pipes = [
        0 => $stream,
        1 => $this->temp,
        2 => array("file", "/dev/null", 'w'),
    ];

    // Launch the process
    $this->cats[] = proc_open($cmd, $pipes, $outputPipes = []);
}

Redirect pipe acquired by proc_open() to file for remainder of process duration

Edit:

3 Answers3