Perl: execute 10 system processes async

Question

This question is for running perl on windows 2012 server.

So I have a folder called Commands_To_Run and under there are 100 batch files e.g.

Commands_To_Run
 - run_1.bat
 - run_2.bat
 - run_3.bat 
...
 - run_100.bat

Each of these run*.bat files take about 30 mins to complete. If I run these batch files serially using a FOR loop then it takes me 100 * 30 min to run. (Too long!)

What I want to is write a perl script that will execute 10 batch files at a time. Once any one of the batch files complete the next batch file would get executed.

For example I would like to execute run1.bat through run10.bat. Let's say run7.bat finishes then I want to run next run11.bat and so on. So there are 10 files running at any given time.

I thought about using this perl script to run batch file but this will run all 100 at the same time and it will kill my windows CPU & processing.

for ($x=0; $x < scalar(@files); $x++ ) {
    $file=@files[$x];
    chomp $file;
    $cmd="start $file ";
    print "Runnung Command is: $cmd\n";
    system($cmd);
}

I looked at the suggestion given but there is no working example of how to use use Forks::Super

Possible duplicate of [perl process queue](https://stackoverflow.com/questions/3424778/perl-process-queue) — pilcrow, Feb 18 '19 at 22:14
@pilcrow I looked at the suggestion given but there is no working example of how to use use Forks::Super. How would it work in my case? I don't get it — Sam B, Feb 18 '19 at 22:23
As an unrelated note, your for loop example can be written more perlish as: `foreach my $file (@files) {...}` (note foreach and for are in practice synonyms) — Grinnz, Feb 19 '19 at 00:26

Grinnz · Answer 1 · 2019-02-18T22:54:39.707

3

The fmap_scalar function from Future::Utils can handle all of the logic of keeping a certain amount of processes running, and IO::Async::Process can run and manage each process asynchronously (given it's windows, I'm not sure if all of this will work sensibly):

use strict;
use warnings;
use IO::Async::Loop;
use Future::Utils 'fmap_scalar';

my @commands = ...;

my $loop = IO::Async::Loop->new;

my $f = fmap_scalar {
  my $cmd = shift;
  my $f = $loop->new_future;
  $loop->open_process(command => $cmd, on_finish => sub { $f->done($_[1]) });
  return $f;
} foreach => \@commands, concurrent => 10;

my @exit_codes = $f->get; # starts the loop until all processes are done

edited Feb 18 '19 at 22:54

answered Feb 18 '19 at 22:23

Grinnz

9,093
11
18

I have strawberry installed and this is what I get -- Can't locate IO/Async/Loop.pm in @INC (you may need to install the IO::Async::Loop module) – Sam B Feb 18 '19 at 22:37
IO::Async and Future::Utils are both non-core modules, so need to be installed. – Grinnz Feb 18 '19 at 22:54

zdim · Accepted Answer · 2019-02-19T18:29:46.503

3

A simple way to run processes in parallel and in a queue is with Parallel::ForkManager

use warnings;
use strict;
use feature 'say';

use Parallel::ForkManager;    

my $pm = Parallel::ForkManager->new(10); 

# Prepare the list of your batch files (better get names from disk)
my @batch_files = map { "Commands_To_Run/run_$_.bat" } 1..100;

foreach my $batch_file (@batch_files)
{
    $pm->start and next;
    # Run batch job
    say "Running: $batch_file";
    #system($batch_file);        # uncomment to actually run the jobs
    $pm->finish;
}
$pm->wait_all_children;

This is a minimal but working script. See, for example, this post and this post for more on how jobs go and in particular on how to return data from jobs.

Note: This is not a core module, so you'd likely need to install it

edited Feb 19 '19 at 18:29

answered Feb 18 '19 at 22:47

zdim

64,580
5
52
81

good working script - for completeness add system($batch_file); after running line as that will do the actual execution of my run_*.bat files – Sam B Feb 18 '19 at 23:12
@SamB Thank you for edits, I've tweaked them a little for what I think you meant. I commented out the line to actually run the jobs, as I always do; I like to leave it to people to check things before they enable their actual bulk runs! Another fluid part is where the folder is in relation to the script, etc, but you can adjust that. – zdim Feb 18 '19 at 23:25
One thing needs to be mentioned here it waits for all children i.e. it will process the next 10 only after every file in the first 10 batch is processed. Sorry I accepted the answer too soon. – Sam B Feb 18 '19 at 23:35
@SamB Um, no it doesn't -- `wait_all_childern` only tells it to "reap" all child processes _after it's all done_ (as all code that forks must; on windows it's probably going to `join` threads). While it's working it starts a new process as soon as one exits; it always keeps it at 10. (Can't see that with just printing because it's so fast!) – zdim Feb 18 '19 at 23:36
let me test it out more thoroughly. thank you for the great answer though. – Sam B Feb 18 '19 at 23:38
@SamB By all means, of course, please! :) Also see links if you wish more explanation, and let me know if yet more would be useful – zdim Feb 18 '19 at 23:39
@SamB For instance, the first link in my answer ([this one](https://stackoverflow.com/a/41891334/4653379)) has an example that shows far more on how the jobs come and go -- you can just copy-paste it I think. (Also, [this post](https://stackoverflow.com/a/39071940/4653379) has _a lot_ more detail on how the module works.) – zdim Feb 18 '19 at 23:45

ikegami · Answer 3 · 2019-02-19T07:35:10.960

Parallel::ForkManager relies on fork, a feature of Unix systems that is badly emulated by Perl (using threads) on Windows systems. I would recommend using threads directly instead. Less can go wrong that way.

use threads; 
use Thread::Queue 3.01 qw( );

sub worker {
   my ($command) = @_;
   system($command);
}

{
   my $q = Thread::Queue->new();
   for (1..10) {
      async {
         while (my $job = $q->dequeue()) {
            worker($job);
         }
      }
   }

   $q->enqueue($_) for @commands;
   $q->end();
   $_->join for threads->list;
}

Gerhard · Answer 4 · 2019-02-19T07:38:50.253

Just putting it out there, but it could be possible doing this with a batch file as well, this should loop through all .bat files, checking process count and only kick of new ones if processes are not less or equal to 9 (if equal to 9 it will still kick of one):

@echo off
setlocal enabledelayedexpansion
set cnt=1
for %%i in (*.bat) do (
    set id=%%i
    call :check
)

:check
for /f "tokens=1,*" %%a in ('tasklist /FI "WINDOWTITLE eq _process*" ^| find /I /C "cmd.exe"') do set procs=%%a
    if !procs! leq 9 (
    if not "!id!"=="%0" start "_process!cnt!" !id!
    set /a cnt+=1
   ) else (
     goto check
 )

Perl: execute 10 system processes async

4 Answers4