Add parallelism to perl script

Question

I have small perl script which gets services details from mongoDB, queries its statuses and gives html output

#...some stuff to get $token

my @cmd = ('/opt/mongo/bin/mongo', '127.0.0.1:27117/service_discovery', '--quiet', '-u', 'xxx', '-p', 'xxx', '--eval', "var environ='$env'; var action='status'", '/home/mongod/www/cgi/getstatus.js');
my $mongo_out;
run \@cmd, '>>', \$mongo_out;
$json->incr_parse ($mongo_out);
while (my $obj = $json->incr_parse) {
    my $hostname = "$obj->{'hostname'}";
    print "<tr><td colspan=4 align=\"center\"><h4>$hostname</h4></td></tr>";
    foreach my $service (@{$obj->{'services'}}) {
            my $name = "$service->{'name'}";
            my $port = "$service->{'port'}";
            my $proto = "$service->{'proto'}";
            my $request = HTTP::Request->new(GET => "${proto}://$hostname:${port}/status/service");
            $request->header(Authorization => "Bearer $token");
            my $ua = LWP::UserAgent->new;
            $ua->timeout(2);
            my $response = $ua->request($request);
            my $code = $response->code();
            if ($code == 200) {
                    my $var = %$response->{'_content'};
                    my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
                    my $out = try {my $output = $coder->decode($var)} catch {undef};
                    if(exists $out->{'name'} && exists $out->{'version'}) {
                            print "<tr><td align=\"center\">$port</td><td align=\"center\">$name</td><td align=\"center\">$out->{'name'}</td><td align=\"center\">$out->{'version'}</td></tr>";
                    } else {
                            print "<tr><td align=\"center\">$port</td><td align=\"center\">$name</td><td colspan=2 align=\"center\">auth failed</td></tr>";
                    }
            } elsif ($code == 500) {
                            print "<tr><td align=\"center\">$port</td><td align=\"center\">$name</td><td colspan=2 align=\"center\">offline</td></tr>";
            } elsif ($code == 404) {
                    print "<tr><td align=\"center\">$port</td><td align=\"center\">$name</td><td colspan=2 align=\"center\">page not found</td></tr>";
            }
    }
}

It executes for a while, especially when some services are offline. Is it possible to query services within same host simultaneously?

A LWP::UserAgent instance can't do concurrent requests, but you can use multiple instances of LWP::UserAgent concurrently in different threads or processes. (Even in Coro threads with the help of LWP::Protocol::AnyEvent::http.) If you want to do everything in one thread, then you can use Net::CURL::Multi. (Bonus: It has far less overhead than LWP) — ikegami, Feb 11 '16 at 16:51

score 5 · Accepted Answer · edited May 23 '17 at 11:51

This is almost a question that's too broad to answer, because ... it depends.

But yes. You have two and a half mechanism for parallelising in perl:

thread
fork
Non blocking IO.

I say two and a half, because non-blocking IO isn't really parallel, as much as solving the same problem a different way.

Implementation of parallelism is a really good way to end up with some horrific and hard to trace bugs, and requires a bit of a shift of mind set, because your code is no longer executing in a well defined sequence - the whole point is that your code might hit different bits at different times, and that can cause utter chaos.

And not least because modules you import - might well not be "thread safe" (which means they may be fine, but occasionally will break in a very unpredictable way, and you'll tear your hair out trying to track down the bug).

So with that in mind

threads

Perhaps slightly counter intuitively, if you've used threads in another language - perl threads are NOT light weight. There is a significant cost to starting them, not least because you effectively end up multiplying your memory footprint by the number of threads you are running.

I would normally suggest as a result - look at a "worker threads" model, using Thread::Queue. You start up a number of threads, and use queues to serialise the input and output from the threads.

forking

fork() is a unix native system call. You use it a lot, and it's quite efficient. It splits your program into two identical copies - including position within the code - at the point at which it's called. The only difference initially is the return code of the fork() system call - the parent will get the process ID of the child, the child will get zero.

It's quite easy to do strange thing accidentally, as both piece of code at this point are at exactly the same point in terms of loop iterations, file handles, etc. but this rapidly diverges and you can again, end up with some very strange things happening if you interact with 'shared' resources.

I would normally suggest looking at Parallel::ForkManager module as an easy way to avoid tripping yourself up with fork().

non blocking IO

You can often use something like IO::Select and the can_read method, which detects which file handles will block if you read from them - you can skip that one, until it blocks. This would also work for your use case, although it's not always applicable.

I've got examples of both the above here: Perl daemonize with child daemons

Add parallelism to perl script

1 Answers1

threads

forking

non blocking IO