0

I can't make parallel connections changing the user agent and passing through a proxy. This is my simple script:

use HTTP::Request; 
use LWP::ConnCache;
use LWP::Parallel::UserAgent;

$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; 


my $reqs = [  
     HTTP::Request->new('GET', 'https://website1.com'),
     HTTP::Request->new('GET', 'https://website2.com'),
     HTTP::Request->new('GET', 'https://website3.com'),
     HTTP::Request->new('GET', 'https://website4.com'),
     HTTP::Request->new('GET', 'https://website5.com'),
     HTTP::Request->new('GET', 'https://website6.com'),
];



my ($req,$res);

# register requests
foreach $req (@$reqs) {
    print "Registering '".$req->url."'\n";

    $ua = LWP::Parallel::UserAgent->new();
    $ua->duplicates(0);
    $ua->timeout(30);        
    $ua->redirect(1);
    $ua->agent("Test Service v1");  #this doesn't work

    my $proxy_server = '192.168.10.10:8080';
    $ua->proxy(['https', 'http', 'ftp'] => $proxy_server);

    $ua->register ($req , \&handle_answer);
}

my $entries = $ua->wait();




sub handle_answer {
    my ($content, $response, $protocol, $entry) = @_;

    print "Handling answer from '",$response->request->url,": ",
          length($content), " bytes, Code ",
          $response->code, ", ", $response->message,"\n";

    if (length ($content) ) {
        $response->add_content($content);
    } else {

    }


    return undef;
}

The only way I found to change the User-Agent is by the module HTTP::Headers and changing the http request.

use HTTP::Headers;
....
...
my $headers = new HTTP::Headers(
    'User-Agent' => "Test Service v1",
); 

my $reqs = [  
     HTTP::Request->new('GET', 'https://website1.com', $headers),
     ...
];

...

But I can't make the requests through the proxy... Thanks for your help

marine
  • 1

1 Answers1

0

Perhaps there are better solutions, but I would use threads:

#!/usr/bin/perl

use threads;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new();
   $ua->agent("MY USER AGENT");
   $ua->proxy(['https', 'http', 'ftp'] => "http://192.168.10.10:8080");

my @urls = (
 'https://website1.com',
 'https://website2.com',
 'https://website3.com'
);

my @threads;
for (@urls) {
   push @threads, async { $ua->get($_) };
}

for my $thread (@threads) {
   my $response = $thread->join;
   if ($response->is_success) {
       print $response->status_line, "\n";
   }
}
  • Thank you for your reply. I already tried with multi threads before, exactly like you wrote here, but the use of memory and cpu is very high, instead the LWP::Parallel seems work much better – marine Oct 29 '16 at 11:00