Mojolicious REST + IOLoop concurrent service

Question

First of all I'm a newbie in Perl and Mojo so I'll do my best with this.. :D

The problem is that I need to work in parallel, so this service need to process several calls every second. This service will allow other services to connect over REST in order to download and process files. I'm doing some tests over this idea without success.

I've been testing around Mojo::IOLoop but I think I've misunderstood some concepts about that because every implementation using loops and timers goes absolutely wrong...

Actually my workflow goes over here:

(1) Get REST call

(2) Verify url provided over mongodb

(3) Download file associed to that mongo object.

So actually this is working but I can't see how to implement this in order to work in parallel...

#!/usr/bin/env_perl
use Mojolicious::Lite;
use MongoDB;

my $mongo     = MongoDB::MongoClient->new(host => 'xxxx.mongohq.com',port => 'xxxxx',     username => 'xxxxx', password => 'xxxxx', db_name => 'xxxxx');
my $database   = $mongo->get_database( 'xxxxx' );
my $collection = $database->get_collection( 'xxxxx' );

any [qw/ get post /] => '/:action/:objid' => sub {

  my $self = shift;

  #Parsing objectid...
  my $objectid  = $self->param('objid');
  my $id = MongoDB::OID->new(value => $objectid);

  my $all = $collection->find({_id => $id});
  my $dts = $all->next;

  download($dts->{parsedoc}->{url},"final_name.xml");

};

app->start;

sub download {
  say "Starting new download..";  
  my $ua = Mojo::UserAgent->new(max_redirects => 5);
  my $tx = $ua->get($_[0]);
  $tx->res->content->asset->move_to($_[1]);
  say "Download ends."; 
}

Which is the best way to implement IOLoop over here? I've been using some mojo timer examples but maybe I need to reorder my app flow?? Thanks in advance.

IOLoop is not about working in parallel but working non-blocking. As such you need a non-blocking Mongo client. I know that the author of Mojolicious has been writing one called Mango, but it is in an alpha state at this point. — Joel Berger, Jul 02 '13 at 01:21

score 2 · Accepted Answer · answered Jul 02 '13 at 01:43

As I mention in my comment, Mojo::IOLoop doesn't concern with parallelism but with event loops. This means that operations which may take a long time may be started and then continued only once complete without blocking the other processes. In your example both the call to Mongo and the download are these types of long-running processes which one may want to put in the event loop. Mojolicious::UserAgent is non-blocking when used with a callback (the action called once the operation is complete). To do the same with Mongo you will need to be sure your Mongo library can be used in a non-blocking fashion.

I have mocked up a little example which ignores the Mongo part (exercise for the reader) and uses the UserAgent in a non-blocking fashion.

#!/usr/bin/env perl
use Mojolicious::Lite;

helper download => sub {
  my ($c, $url, $target) = @_;
  my $ua = $c->app->ua->max_redirects(5);

  say "Starting new download ($url) ..."; 
  $ua->get( $url, sub {
    my ($ua, $tx) = @_;
    say "Finished download ($url) ...";
    $tx->res->content->asset->move_to($target);

    say "File saved ($url --> $target)";
    $c->render( text => "File saved ($url --> $target)" );
  });
};

any [qw/ get post /] => '/:action/:objid' => sub {
  my $self = shift;
  $self->render_later; # prevent auto-render

  #Parsing objectid
  my $objectid  = $self->param('objid');
  #verify url here ...

  # example url
  my $url = 'http://www.google.com';

  $self->download($url,"final_name.xml");
};

app->start;

Thanks Joel I think that's the point. Actually I was lost with Loop concept in Perl. I've been testing your sollution and works fine despite some inactivity timeouts (I think that comes from the "deferred" render during large downloads). — Courier, Jul 02 '13 at 09:37
The first iteration of that code just rendered `download started` and left the download to continue on the server side. Thats one possibility. Another is to open a websocket to the client to keep it informed about the progress. Anyway, thats all implementation. :-) — Joel Berger, Jul 02 '13 at 11:21

Mojolicious REST + IOLoop concurrent service

1 Answers1