How to run multiple perl open commands asynchronously and display output in order

Question

I'm attempting to run multiple SSH commands to multiple servers asynchronously, I'd like to capture the output from the commands and display them in order. To throw in an additional curveball, I'd like pid3 to only run once pid2 is complete, and pid4 to run after the first three commands complete. How would this best be accomplished?

Example:

// $pid1 and $pid2 should run asynchronously
my $pid1 = open(my $SSH1, "|ssh -t -t runuser\@$server{'app'} 'sudo chef-client'");

my $pid2 = open(my $SSH2, "|ssh -t -t runuser\@$server{'auth'} 'sudo chef-client'");

// This command should wait for $pid2 to complete.
my $pid3 = open(my $SSH3, "|ssh -t -t runuser\@$server{'auth'} \"sudo -- sh -c '$update_commands'\"");

// This command should wait for $pid1-3 to complete before running.
my $pid4 = open(my $SSH4, "|ssh -t -t runuser\@$server{'varn'} \"sudo -- sh -c '$varn_commands'\"");

See the answer by tsee in http://stackoverflow.com/questions/1752357/how-can-i-run-a-system-command-in-perl-asynchronously?rq=1 . Also, the only asynchronous stuff in your question is pid1 vs pid2/3. Everything else is sequential. — imran, Apr 05 '13 at 04:31
You're correct in the desired effect, although currently all 4 commands run asynchronously. — Highway of Life, Apr 05 '13 at 04:33
Before you create $pid3, make sure you process and close $SSH2. Then before you create $pid4, make sure you process and close $SSH1 and $SSH3. But there are better ways than using open(), see the suggestions in the link above. — imran, Apr 05 '13 at 04:39

score 2 · Answer 1 · answered Apr 05 '13 at 15:51

Forks::Super handles all of these requirements:

use Forks::Super;

# run  $command1  and  $command2 , make stderr available
my $job1 = fork { cmd => $command1, child_fh => 'err' };
my $job2 = fork { cmd => $command2, child_fh => 'err' };

# job 3 must wait for job 2. Collect stdout, stderr
my $job3 = fork { cmd => $command3, depend_on => $job2, child_fh => 'out,err' };

# and job 4 waits for the other 3 jobs
my $job4 = fork { cmd => $command4, depend_on => [ $job1, $job2, $job3 ],
                  child_fh => 'out,err' };

# wait for jobs to finish, then we'll collect output
$_->wait for $job1, $job2, $job3, $job4;
my @output1 = $job1->read_stderr;
my @output2 = $job2->read_stderr;
my @output3 = ($job3->read_stdout, $job3->read_stderr);
my @output4 = ($job4->read_stdout, $job4->read_stderr);
...

score 2 · Answer 2 · answered Apr 07 '13 at 21:51

2

Use Net::OpenSSH::Parallel:

# untested!
use Net::OpenSSH::Parallel;
my $pssh = Net::OpenSSH::Parallel->new;

$pssh->add_server(app  => $server{app},  user => 'runuser');
$pssh->add_server(auth => $server{auth}, user => 'runuser');
$pssh->add_server(varn => $server{varn}, user => 'runuser');

$pssh->push('app',  cmd  => @cmd1);
$pssh->push('auth', cmd  => @cmd2);
$pssh->push('auth', cmd  => @cmd3);
$pssh->push('varn', join => '*');
$pssh->push('varn', cmd  => @cmd4);

$pssh->run;

Automating sudo is slighly more complex if you need to pass the passwords, but still can be done. It is explained on the module docs.

answered Apr 07 '13 at 21:51

salva

9,943
4
29
57

In my case, I don't have to pass the passwords as I'm using SSH Key's. – Highway of Life Apr 09 '13 at 00:23
@HighwayofLife: I was not referring to ssh authentication. Usually, when you run some command with `sudo` it asks you for an additional password. Though in your environment, `sudo` may well be configured to not ask for passwords. – salva Apr 09 '13 at 07:46
Ah, yes... and you'd be correct. My sudo user is part of the wheel group, which by default doesn't require a password to run sudo commands. But good point for others who may encounter that problem. – Highway of Life Apr 11 '13 at 03:21

score 1 · Accepted Answer · answered Apr 05 '13 at 04:51

My (somewhat crude) solution thus far. I feel there may be a more elegant way to handle this in Perl, but this may get the job done:

# Silence all non-error output from the commands on first 2 servers:
my $pid1 = open(my $SSH1, "|ssh -t -t runuser\@$server{'app'} 'sudo chef-client > /dev/null'");

my $pid2 = open(my $SSH2, "|ssh -t -t runuser\@$server{'auth'} 'sudo chef-client > /dev/null'");

if ($pid1) {
    print "Connecting to $server{'app'}: chef-client";
    while ( <$SSH1> ) {
        print $server{'app'};
        print $_;
    }
}
close $SSH1 or die $!;

if ($pid2) {
    print "Connecting to $server{'auth'}: chef-client";
    while ( <$SSH2> ) {
        print $server{'auth'};
        print $_;
    }
}
close $SSH2 or die $!;

# Run pid3 once pid2 is closed
my $pid3 = open(my $SSH3, "|ssh -t -t runuser\@$server{'auth'} \"sudo -- sh -c '$update_command'\"");
if ($pid3) {
    print "Connecting to $server{'auth'}: $update_command";
    while ( <$SSH3> ) {
        print $_;
    }
}
close $SSH3 or die $!;

# Run pid4 after all previous commands have completed.
my $pid4 = open(my $SSH4, "|ssh -t -t runuser\@$server{'varn'} \"sudo -- sh -c '$varn_command'\"");
if ($pid4) {
    print "Connecting to $server{'varn'}: $varn_command";
    while ( <$SSH4> ) {
        print $_;
    }
}
close $SSH4 or die $!;

How to run multiple perl open commands asynchronously and display output in order

3 Answers3