I need to perform a large number of HTTP post requests, and ignore the response. I am currently doing this using LWP::UserAgent. It seems to run somewhat slow though I am not sure if it is waiting for a response or what, is there anyway to speed it up and possibly just ignore the responses?
Asked
Active
Viewed 3,142 times
3
-
I don't know `LWP::UserAgent` that well, but have you tried setting the `max_size` to a low number? Perhaps the transfer is interrupted once the `max_size` is reached? – Gaurav Apr 28 '11 at 18:07
2 Answers
4
bigian's answer is probably the best for this, but another way to speed things up is to use LWP::ConnCache to allow LWP to re-use existing connections rather than build a new connection for every request.
Enabling it is this simple if you're pounding on just one site --
my $conn_cache = LWP::ConnCache->new;
$conn_cache->total_capacity([1]) ;
$ua->conn_cache($conn_cache) ;
I've found this to double the speed of some operations on http site, and more than double it for https sites.
Doug McLaren
- 41
- 2
2
LWP::Parallel
http://metacpan.org/pod/LWP::Parallel
"Introduction
ParallelUserAgent is an extension to the existing libwww module. It allows you to take a list of URLs (it currently supports HTTP, FTP, and FILE URLs. HTTPS might work, too) and connect to all of them in parallel, then wait for the results to come in."
It's great, it's worked wonders for me...
