$_ empty while reading STDIN in parallel

Question

I have a legacy project which take a huge amount of data from STDIN and process it line by line in a perl script. The line order is not important. This is taking very long so I want to make it in parallel.

After a bit of research I found Parallel::Loops which seems suitable but I can't get it working because $_ is empty. My code is:

#Initialize all vars etc

$pl->while ( sub { <STDIN> }, sub {
    print $_       # but $_ is empty
}

Other ways of reading from STDIN ir parallel are wellcome too.

---

Update:

After all the help that I received I could manage some working piece of code, thank you. I'm going to do a brief abstract. To clarify:

1. This is a kind of parser, it has more than 3000 lines with regex and conditions which were auto generated.

2. The input that I use for testing is a POS tagged text, there are 1071406 lines in this file.

3. My hardware is: SSD disc, mid range i5 last gen and 8gb RAM DDR4.

Conclusions:

1. As the comments suggested IO operations make my script slow.
2. All the suggestion resulted in improvements, specially the ones including processing bunch of lines instead of line by line.
3. Answers contain very useful implementation of threading for future work.
4. Framework Parallel::ForkManager introduce a lot of lag in the execution time. I always kill the script after 5min, since the script without parallelism takes about 6.
5. Framework Parallel::Loops introduce a little improvement. The script takes about 3min to finish.
6. Using GNU parallel is the easy way of optimizing.
7. Using the package Threads I got the best time, 1min45secs but it is very close to GNU parallel so it's on you giving it a try, and making the effort of porting the code.
8. Using the thread package as in the @ikegami answer reading bunch of lines, times were the same as the times got applying the @tanktalus solution, which read line by line.

Finally, I'm going with the @ikegami solution, which I think that will be better when the amount of data increase. I adjust the amount of lines to process to 100.000 because it gets better results than 10.000, for instance. This difference is matter of 8 secs aprox.

Next natural step is writting everything to files instead of using the STDOUT, I hope this helps to reduce the time a little bit more.

Answer 1

$_ is never set because you never assign to $_!

Don't forget that

while (<STDIN>) { ... }

is short for

while (defined( $_ = <STDIN> )) { ... }

That means you were looking to use the following:

$pl->while ( sub { defined( $_ = <STDIN> ) }, sub {
    print $_;
}

That said, clobbering $_ is a bad idea. It could very well have been aliased to some other variable by a for (...) in the caller.

That means you should be using the following:

my $line;
$pl->while ( sub { defined( $line = <STDIN> ) }, sub {
    print $line;
}

You may find that breaking down the work into coarser units that lines will yield better performance as it reduces the overheard to work ratio.

use constant WORK_UNIT_SIZE => 100;

my $done = 0;
my @lines;
$pl->while ( sub {
    @lines = ();
    return 0 if $done;

    while (@lines < WORK_UNIT_SIZE) {
        my $line = <>;
        if (!defined($line)) {
            $done = 1;
            return 0+@lines;
        }

        push @lines, $line;
    }

    return 1;
}, sub {
    for (@lines) {
        print $_;
    }
}

Finally, rather than creating a new task for each work unit, you should reuse them! The following demonstrates this using threads.

use threads            qw( async );
use Thread::Queue 3.01 qw( );

use constant NUM_WORKERS    => 8;
use constant WORK_UNIT_SIZE => 100;

sub worker {
    my ($job) = @_;
    for (@$job) {
        print $_;
    }
}

my $q = Thread::Queue->new();
$q->limit(NUM_WORKERS * 4);

async { while (defined( my $job = $q->dequeue() )) { worker($job); } }
    for 1..NUM_WORKERS;

my $done = 0;    
while (!$done) {
    my @lines;
    while (@lines < WORK_UNIT_SIZE) {
        my $line = <>;
        if (!defined($line)) {
            $done = 1;
            last;
        }

        push @lines, $line;
    }

    $q->enqueue(\@lines) if @lines;
}

$q->end();
$_->join for threads->list;

Answer 2

I don't know about specific benefits to using Parallel::Loops (which there may well be). Here is the same with Parallel::ForkManager, which is what Parallel::Loops uses.

use warnings;
use strict;
use feature 'say';

use Parallel::ForkManager;   

my $max_procs = 30; 
my $pm = Parallel::ForkManager->new($max_procs);   

# Retrieve data returned by children in the callback
my %ret_data;      
$pm->run_on_finish( sub { 
    my ($pid, $exit, $ident, $signal, $core, $dataref) = @_; 
    $ret_data{$pid} = $dataref;
});

while (my $input = <STDIN>)
{
    chomp($input);

    $pm->start and next;
    my $ret = run_job($input);
    $pm->finish(0, \$ret);
}
$pm->wait_all_children;

foreach my $pid (keys %ret_data) {
    say "$pid returned: ${$ret_data{$pid}}";
}

sub run_job { 
    my ($input) = @_; 
    # your processing
    return $input;    # to have something to check
} 

This code returns a scalar from a child process, a single value. You can return any data structure, see Retrieving data structures from child processes in docs and this post for an example.

Data is returned via files and that can slow things down for large data or many quick processes.

If testing at a terminal stop input with Ctrl-d (or add last if $input !~ /\S/; after chomp to stop with empty line -- but not with data being passed to STDIN by other means).

---

It is clarified that each STDIN read is just one line to process. Then we should collect more lines before spawning a new process, otherwise there is way too much overhead.

my $num_lines_to_collect = 1000;

my @lines_to_process;         # collect lines for each fork

while (my $input = <STDIN>)
{
    chomp($input);
    push @lines_to_process, $input;
    next if $. % $num_lines_to_collect != 0;

    $pm->start and next;
    my $ret = run_job( \@lines_to_process );
    $pm->finish(0, \$ret);

    @lines_to_process = ();   # empty it for the next round
}
$pm->wait_all_children;

We add lines to the array @lines_to_process and proceed to triger a new fork only when the current line number $. is a multiple of $num_lines_to_collect. So a job is started for every $num_lines_collect, so each job processes that much. I set it to 1000, experiment.

Answer 3

Probably the simplest manner here is to create a pool of threads, each one listening on the same queue, and then have one thread (probably the main thread) read the file and push each line on to the queue.

use strict;
use warnings;
use Thread qw(async);
use Thread::Queue;

my $q = Thread::Queue->new();
$q->limit(32); # no point in reading in more than this into memory.

my @thr = map {
    async {
        while (defined (my $line = $q->dequeue()) ) {
            print $line;
        }
    };
} 1..4; # 4 worker threads

while (<STDIN>)
{
    $q->enqueue($_);
}
$q->end();

$_->join for Thread->list;

Just as a point of warning, beware if you need to push data from the worker threads back to the main thread. It's not as trivial as in other languages.

Update: switch from threads to Thread. While the async function is documented as returning thread objects, that didn't seem to work for me, so had to change the join as well.