10

I have some Perl code that executes a shell script for multiple parameters, to simplify, I'll just assume that I have code that looks like this:

for $p (@a){
    system("/path/to/file.sh $p&");
}

I'd like to do some more things after that, but I can't find a way to wait for all the child processes to finish before continuing.

Converting the code to use fork() would be difficult. Isn't there an easier way?

Chris
  • 1,416
  • 18
  • 29
Osama Al-Maadeed
  • 5,654
  • 5
  • 28
  • 48
  • Do you require the actual return code from file.sh? Can you change the script so that it writes to a file when complete? – Matt K May 26 '09 at 16:18
  • 1
    Why would converting the code to fork be difficult? Hide all the details in a subroutine. You could even redefine system() to call your new subroutine instead. – brian d foy May 26 '09 at 16:35
  • I'll probably just split this part to a separate script that does the forking, and call that from the system call... – Osama Al-Maadeed May 26 '09 at 17:51

3 Answers3

18

Using fork/exec/wait isn't so bad:

my @a = (1, 2, 3);
for my $p (@a) {
   my $pid = fork();
   if ($pid == -1) {
       die;
   } elsif ($pid == 0) {
      exec '/bin/sleep', $p or die;
   }
}
while (wait() != -1) {}
print "Done\n";
Dave
  • 10,369
  • 1
  • 38
  • 35
15

You are going to have to change something, changing the code to use fork is probably simpler, but if you are dead set against using fork, you could use a wrapper shell script that touches a file when it is done and then have your Perl code check for the existence of the files.

Here is the wrapper:

#!/bin/bash

$*

touch /tmp/$2.$PPID

Your Perl code would look like:

for my $p (@a){
    system("/path/to/wrapper.sh /path/to/file.sh $p &");
}
while (@a) {
    delete $a[0] if -f "/tmp/$a[0].$$";
}

But I think the forking code is safer and clearer:

my @pids;
for my $p (@a) {
    die "could not fork" unless defined(my $pid = fork);\
    unless ($pid) { #child execs
        exec "/path/to/file.sh", $p;
        die "exec of file.sh failed";
    }
    push @pids, $pid; #parent stores children's pids
}

#wait for all children to finish
for my $pid (@pids) {
    waitpid $pid, 0;
}
Chas. Owens
  • 64,182
  • 22
  • 135
  • 226
  • This is a much better answer. – zdim Sep 23 '16 at 18:48
  • Apparently doing this creates a lot of zombie processes. https://en.wikipedia.org/wiki/Zombie_process – Pratyush Rathore Jan 04 '18 at 16:54
  • 1
    @PratyushRathore It shouldn't. A zombie results when the parent process doesn't call wait or waitpid. As you can see the parent is calling waitpid for a child that has exited. If you are getting a bunch of zombies then either you are doing something wrong or one of the first children is taking a long time and the later children have exited (but not been reaped). You could switch to `while (waitpid(-1, WNOHANG) > 0) { sleep 1 }` to reap the next exited child. – Chas. Owens Jan 05 '18 at 17:41
  • "one of the first children is taking a long time and the later children have exited (but not been reaped)." Yes, that is exactly what was happening in my case. Obviously, this was a very very special case with a lot of processes. I think, I wrote the last comment in a pretty bad style. I just wanted to ensure that people have a context for zombie processes when they are doing this. – Pratyush Rathore Jan 07 '18 at 15:50
9

Converting to fork() might be difficult, but it is the correct tool. system() is a blocking call; you're getting the non-blocking behavior by executing a shell and telling it to run your scripts in the background. That means that Perl has no idea what the PIDs of the children might be, which means your script does not know what to wait for.

You could try to communicate the PIDs up to the Perl script, but that quickly gets out of hand. Use fork().

darch
  • 4,200
  • 1
  • 20
  • 23