Mojo useragent, create connection pool

Question

Trying to limit the number of connections to a server when using non blocking requests. The idea is that I have many sub pages on the same host that I query very often. I would like to open a fixed number of connections and reuse them until all requests are done.

In my example I'm using 10 sub pages on the same host. Instead of getting all of them at the same time, and open 10 connections to the same host, I would like to do five and as soon as one is done, that connection is reused.

I didn't find any examples of Mojo::UserAgent pooling.

https://pastebin.com/GRtBdYP3

#!/usr/bin/env perl
use Mojolicious::Lite;

use Mojo::UserAgent;
use Mojo::IOLoop;
use Data::Dumper;

# Documentation browser under "/perldoc"
plugin 'PODRenderer';


my @url_lst = (
    'https://stackoverflow.com/questions/1607904/vim-deleting-from-current-position-until-a-space',
    'https://stackoverflow.com/questions/7409134/english-mnemonics-to-vims-shortcuts?noredirect=1&lq=1',
    'https://stackoverflow.com/questions/102384/using-vims-tabs-like-buffers?rq=1',
    'https://stackoverflow.com/questions/327411/how-do-you-prefer-to-switch-between-buffers-in-vim?noredirect=1&lq=1',
    'https://stackoverflow.com/questions/105721/how-do-i-move-to-end-of-line-in-vim?rq=1',
    'https://stackoverflow.com/questions/235839/indent-multiple-lines-quickly-in-vi?rq=1',
    'https://stackoverflow.com/questions/2332340/indenting-a-bunch-of-lines-in-vim?noredirect=1&lq=1',
    'https://stackoverflow.com/questions/657447/vim-clear-last-search-highlighting?rq=1',
    'https://stackoverflow.com/questions/14863315/highlight-searches-in-vim-but-not-substitutions?noredirect=1&lq=1',
    'https://stackoverflow.com/questions/1497958/how-do-i-use-vim-registers?rq=1',
);

my @req_lst;

my $ua = Mojo::UserAgent->new;

Mojo::IOLoop->singleton->recurring(5 => sub {
        my $j = 0;

        #foreach my $j (0..2) {
            foreach my $i (0..4) {
                $req_lst[$i] = $ua->get_p($url_lst[$i+($j*5)]);
                print Dumper($url_lst[$i+($j*5)]);
            }

            Mojo::Promise->all($req_lst[0], $req_lst[1], $req_lst[2], $req_lst[3], $req_lst[4])->then(sub {
                    my (@lst) = @_;

                    #print Dumper(@lst);
                })->wait;
            #}
    }
);



get '/' => sub {
    my $c = shift;
    $c->render(template => 'index');
};

app->start;
__DATA__

@@ index.html.ep
% layout 'default';
% title 'Welcome';
<h1>Welcome to the Mojolicious real-time web framework!</h1>
To learn more, you can browse through the documentation
<%= link_to 'here' => '/perldoc' %>.

@@ layouts/default.html.ep
<!DOCTYPE html>
<html>
  <head><title><%= title %></title></head>
  <body><%= content %></body>
</html>

The reason for doing this is that I'm accessing a server lots of times each second for monitoring. The server only supports five open connections at once, the others are dropped. Since the connection goes through a firewall, each time a new connection is opened to the server, it get logged. By just open five connections, and reuse them, it will limit the logging significantly.

Mojo useragent, create connection pool

0 Answers0