Perl script to log external executables output and error while that still runs

Question

I have a Perl script, which runs an external executable. That executable runs for a while (sometimes seconds, sometimes an hour), can spit out text to both STDOUT and STDERR as well as an exit code, which all are needed. Following code demonstrates first successful external executable run (small bash script with one line - the comment), then with bad exit status (example with gs - ghostscript). I want the external executable give its STDOUT to the Perl script for evaluation, filtering, formatting etc. before it gets logged to a logfile (used for other stuff as well) while the external is still executing. STDERR would also be great to be worked on same way. This script is in stand to log everything from STDOUT, but only after the executable has finished. And the STDERR is logged only directly, without evaluations etc. I have no possibility to install any additional Perl parts, modules etc.

How do I get my Perl script to get each line (STDOUT + STDERR) from the executable while it is spitting it out (not just at the end) as well as its exit code for other purposes?

#!/usr/bin/perl
@array_executable_and_parameters = "/home/username/perl/myexecutable.sh" ; #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
@array_executable_and_parameters2= "gs aaa" ;
my $line;
chdir("/home/username/perl/");
$logFileName = "logfileforsomespecificinput.log";
open(LOGHANDLE, ">>$logFileName" );
open (STDERR, '>>', $logFileName);                  #Prints to logfile directly
#open (STDERR, '>>', <STDOUT>);                 #Prints to own STDOUT (screen or mailfile)

print LOGHANDLE "--------------OK run\n";
open CMD, '-|', @array_executable_and_parameters or die $@;
while (defined($line = <CMD>)) {                    #Logs all at once at end
    print LOGHANDLE "-----\$line=$line-----\n";
}
close CMD;
$returnCode1= $?>>8;
print LOGHANDLE "\$returnCode1=$returnCode1\n";

print LOGHANDLE "--------------BAD run\n";
open CMD2, '-|', @array_executable_and_parameters2 or die $@;
while (defined($line = <CMD2>)) {
    print LOGHANDLE "-----\$line=$line-----\n";
}
close CMD2;
$returnCode2= $?>>8;
print LOGHANDLE "\$returnCode2=$returnCode2\n";

close(LOGHANDLE);

Take 2. After good advice in comments I have tried the IPC::Run. But something still does not work as expected. I seem to be missing how the looping from start (or pump?) to finish works, as well as how to get it to iterate when I do not know what the last output would be - as the examples everywhere mentions. So far I have now the following code, but it does not work line by line. It spits out listing of files in one go, then waits until the external loop is fully finished to print all the X's out. How do I tame it to the initial needs?

#! /usr/bin/perl
use IPC::Run qw( start pump finish );

@array_executable_and_parameters = ();
push(@array_executable_and_parameters,"/home/username/perl/myexecutable.sh"); #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
my $h = start \@array_executable_and_parameters, \undef, \$out, \$err ;
pump $h;# while ($out or $err);
print "1A. \$out: $out\n";
print "1A. \$err: $err\n";
$out = "";
$err = "";
finish $h or die "Command returned:\n\$?=$?\n\$@=$@\nKilled by=".( $? & 0x7F )."\nExit code=".( $? >> 8 )."\n" ;
print "1B. \$out: $out\n";
print "1B. \$err: $err\n";

score 6 · Answer 1 · answered Jan 10 '18 at 13:01

6

Look at IPC modules, especially IPC::Cmd, IPC::Run and if not satisfied then IPC::Run3. There is a lot of details you would have to cover and those modules will make your life a lot easier.

answered Jan 10 '18 at 13:01

Hynek -Pichi- Vychodil

26,174
5
52
73

8

While I agree that this answer would be made more useful with the inclusion of a simple example, *reviewers* please note that the links are to the actual documentation of the modules recommended to solve the issue, not to some transient blog post etc. metacpan is much *much* less likely to change or disappear. – Sinan Ünür Jan 10 '18 at 14:27
1

@SinanÜnür: There are many reasons to avoid a link-only solution other than the volatility of those links, and this should have been posted as a comment. A full answer should include an explanation of the errant behaviour, as well as a reason why any links could make a difference. Clearly it is best of all if sample code is also provided. – Borodin Jan 10 '18 at 17:28
Yes, looks promising. But ... I can not install, use external code, modules, dependencies etc. Isn't there a simple, manual way to achieve same results? – uldics Jan 10 '18 at 21:11
2

@uldics You mean like simpler than reinventing the wheel? If you would know that `open my $fh, '-|', ...` is internally just bunch of calls to `pipe()`, `fork()`, `close()`, `dup()`, `execl()` or `execve()` and so on, you wouldn't have to ask. But you ask so the best advice is to stick to the recommended modules. It's not a simple task and a lot of opportunities for nasty bugs. – Hynek -Pichi- Vychodil Jan 10 '18 at 23:35
IPC::Cmd is unable to interleave STDOUT and STDERR, IPC::Run3 probably waits till finished (does not allow interaction with the subprocess), so from your three suggestions I suppose only one able to satisfy my requirements is IPC:Run. How do I get it to perform like I have asked? I see no examples anywhere with other than file output redirection. Can it do something like open-pipe-while in my example? – uldics Jan 16 '18 at 12:37
@uldics If you would bother to read the documentation of IPC::Run there is an example about commands `start`, `pump`, `finish`, `timeout`. Namely `## Scripting subprocesses (like Expect):` – Hynek -Pichi- Vychodil Jan 17 '18 at 18:25
I have studied IPC::Run "Scripting subprocesses" and poked and pulled in various directions. No luck so far. Almost every example has input, then output. And they assume output will be known (same as last input). I do not know what will be the output: `pump $h until $out =~ /input\n/g;`. Seems to me I do not understand some more basic stuff here - how does the while work here, does it just runs the one line, or all after it until the finish? – uldics Feb 27 '18 at 10:29

score 1 · Answer 2 · answered Mar 02 '18 at 07:18

OK, have got it to work, so far. Might have some issues - not sure about environment variables, like umask or language related or the system load when push is waiting/blocking, or how to replace die with capturing of all variables for status. Nevertheless for my purpose, seems to work well. Will see how it works on a real system.

#! /usr/bin/perl
BEGIN {
    push @INC, '/home/myusername/perl5/lib/perl5';          #Where the modules from Cpan are
}
use IPC::Run qw( start pump finish );

@array_executable_and_parameters = ();
push(@array_executable_and_parameters,"/home/myusername/perl/myexecutable.sh"); #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
my $h = start \@array_executable_and_parameters, \undef, \$out, \$err ;
while (42) {
    pump $h;# while ($out or $err);
    if ($out eq '' and $err eq '') {last;}
    print "1A. \$out: $out\n";
    print "1A. \$err: $err\n";
    $out = "";
    $err = "";
}
finish $h or die "Command returned:\n\$?=$?\n\$@=$@\nKilled by=".( $? & 0x7F )."\nExit code=".( $? >> 8 )."\n" ;
print "1B. \$out: $out\n";
print "1B. \$err: $err\n";

The key was understanding how the blocking of pump works. All the manuals and help places kind of skipped over this part. So a neverending while which jumps out when pump lets go further without output was the key.

Perl script to log external executables output and error while that still runs

2 Answers2