3

I am looking at the AnyEvent::Fork module. I have 20 external scripts I would like to invoke in parallel (6 at a time) and summarize their output later when all finished. I am at loss how to achieve this.

The example code (that only invokes 1 child) in the module has a problem. I just add a simple sleep to the code to not return at once and the parent exits immediately without waiting for the child process.

open my $output, ">/tmp/log" or die "$!";

AnyEvent::Fork
   ->new
   ->eval ('
        # compile a helper function for later use
        sub run {
           my ($fh, $output, @cmd) = @_;

           # perl will clear close-on-exec on STDOUT/STDERR
           open STDOUT, ">&", $output or die;
           open STDERR, ">&", $fh or die;

           ### Added by me to demonstrate that
           ### $cv->recv returns immediately.
           sleep 5;

           exec @cmd;
        }
     ')
   ->send_fh ($output)
   ->send_arg ("/bin/echo", "hi")
   ->run ("run", my $cv = AE::cv);

my $stderr = $cv->recv;

The result is that /tmp/logis empty. I do not get how condvar is used here, it is not in the documentation. Can I get the number of running children using condvar?

Please help how to get this right.

UPDATE the main issue here is that the parent does not wait for the child to complete.

G. Cito
  • 6,210
  • 3
  • 29
  • 42
cstamas
  • 378
  • 2
  • 13
  • (I can replicate the problem, and nothing obvious pops to mind. I don't have time to look further. The fact that the returned value is name `$stderr` is intersesting. Maybe it doesn't signal the end of the process.) – ikegami Nov 26 '14 at 18:05
  • I bet it's failing because you have no function call `run` to run. Sorry, not sure how to use this module, and I don't have time to find out right now – ikegami Nov 26 '14 at 18:20
  • Is this a different API from [`AnyEvent->condvar`](https://metacpan.org/pod/AnyEvent#CONDITION-VARIABLES)? Here running your code I have "hi" appearing `/tmp/log` after 5 seconds. What is the behavior you expect from the test script? – G. Cito Nov 26 '14 at 19:41
  • @G.Cito /tmp/log is empty for me. ikegami confirmed this. – cstamas Nov 26 '14 at 19:53
  • Not sure "how to continue in chat" - but just for info: **perl5-16.3** FreeBSD AnyEvent::Fork 1.2 , AnyEvent 7.07 ... perl compiled with `-Dusethreads=y` ... `for ((i = 0; i < 42; i++)) ; do perl ae-fork.pl ;done` Using your code but switched up `">>/tmp/log"` and `("/bin/echo", "hi from $$")` ... I get a 42 line log file with PIDs – G. Cito Nov 26 '14 at 20:52
  • i am on debian perl 5.20.1-2 amd64 – cstamas Nov 26 '14 at 21:03
  • Brewed up a 5.20.1 install to try and everything worked as above. Since @ikegami confirms the same behaviour you see the obvious errors I can think of that might explain it don't apply (Unix and permissions?). This will get figured out. – G. Cito Nov 27 '14 at 02:14
  • @G. Cito, The problem isn't that the child doesn't get spawned, the question is how do I wait for the child to finish? – ikegami Nov 27 '14 at 13:59
  • @cstamas, You said the program wasn't waiting for the child to complete, and I confirmed *that*. `/tmp/log` isn't empty if you wait 5 seconds. – ikegami Nov 27 '14 at 14:00
  • what did the working version of you code look like? (you could show this in your **EDIT** section). Did you simply replace the `->run()` arguments? – G. Cito Dec 17 '14 at 18:14

1 Answers1

4

The problem here is that parent and child are separate processes, and can run independent of each other, so can if one should wait for the other, explicit synchronisation is required. There are many ways to do it, the simplest would be to use AnyEvent::Fork::RPC, and send, say, a "wait" request to the child, which it answers when it's finished.

To do it with naked AnyEvent::Fork, the easiest way is to take advantage of the bidrectional pipe provided by ->run:

  AnyEvent::Fork
    ->new
    ->run (sub {
       my ($fh) = @_;
       sysread $fh, my $dummy, 1; # will wait for data, or eof
       ... done, you can now call e.g. $cv->send

The sysread will try to read from the child. If the child never sends anything, this will block the parent until the child exits, as in that moment, the child will close it's end of the pipe and sysread gets an EOF.

Of course, in an AnyEvent program you probably don't want to block, so you use an I/O watcher:

 ->run (sub {
    my ($fh) = @_;
    my $rw; $rw = AE::io $fh, 0, sub {
       ... read data received, or EOF
       undef $rw;
       ... done, you can now call e.g. $cv->send;
    }
  });

This trick can be used for external commands, too (exec in your example), by clearing the close-on-exec state of the pipe in the child and thus passing it to the exec'ed program - in this case, when all programs that inherit the pipe exit, the pipe will signal EOF.

This should get you started. There are other ways to do it, but most or even all of the good ones would involve some pipe for communication, and the easiest one to get one is to use the one provided by AnyEvent::Fork.

Remember Monica
  • 3,897
  • 1
  • 24
  • 31