Stack Overflow
Stack Overflow

Questions tagged [lwp]

The World-Wide Web library for Perl - LWP is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web.

The main focus of the library is to provide classes and functions that allow you to write WWW clients. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

Most modules in this library provide an object oriented API. The user agent, requests sent and responses received from the WWW server are all represented by objects. This makes a simple and powerful interface to these services. The interface is easy to extend and customize for your own needs.

The main features of the library are:

  • Contains various reusable components (modules) that can be used separately or together.
  • Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources.
  • Provides a full object oriented interface or a very simple procedural interface.
  • Supports the basic and digest authorization schemes.
  • Supports transparent redirect handling.
  • Supports access through proxy servers.
  • Provides parser for robots.txt files and a framework for constructing robots.
  • Supports parsing of HTML forms. Implements HTTP content negotiation algorithm that can be used both in protocol modules and in server scripts (like CGI scripts).
  • Supports HTTP cookies.
  • Some simple command line clients, for instance lwp-request and lwp-download.

More information on the CPAN website.

365 questions
3
votes
3 answers

Image corruption using LWP::UserAgent

The following script runs in a loop, retrieving images using LWP::UserAgent, and resizing them using Image::Magick. I am getting this error from Image::Magick when reading the downloaded image: Exception 450: Unsupported marker type 0x54 If I…
chrisrth
  • 1,182
  • 3
  • 15
  • 36
3
votes
2 answers

Why is Perl HTTP::Response not decoding this apostrophe?

I'm using my $ua = new LWP::UserAgent; $ua->agent("Mozilla/5.0 (Windows NT 6.1; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1 "); my $url = "http://somedomain.com/page/"; my $req = new HTTP::Request 'GET' =>…
Ivy
  • 887
  • 1
  • 7
  • 25
3
votes
2 answers

Can can I encode spaces as %20 in a POST from WWW::Mechanize?

I'm using WWW::Mechanize to do some standard website traversal, but at one point I have to construct a special POST request and send it off. All this requires session cookies. In the POST request I'm making, spaces are being encoded to + symbols,…
aidan
  • 9,310
  • 8
  • 68
  • 82
3
votes
2 answers

Get raw response headers from LWP?

Is there a way to grab raw, unmodified response headers from an HTTP request made with LWP? This is for a diagnostic tool that needs to identify problems with possibly malformed headers. The closest thing I've found is: use LWP::UserAgent; my $ua…
plasticinsect
  • 1,702
  • 1
  • 13
  • 23
3
votes
1 answer

Perl transparent proxy

I'm trying to create one transparent HTTP proxy. It's purpose is to stay between the browser and the web server and be invisible. Here is the code I'm using. Unfortunately it's not working very well. When I open the web page (referenced by…
user551761
3
votes
3 answers

While using LWP:UserAgent, $response->is_success returns blank

I am using LWP::UserAgent like below my $ua = LWP::UserAgent->new; my $response = $ua->request ( ...); if ( $response->is_success() ) { ... } print $response->is_success(); Problem I am facing is that is_success() is returning blank. I was…
user966588
3
votes
4 answers

Perl LWP::UserAgent hangs for 120 seconds seemingly randomly against a given server

I noticed a delay of approximately 120 seconds between handling the response_data and response_done events in WWW::Mechanize with a given https web site. I checked with a normal web browser and do not experience this slowness so I suspect there is…
Benoit
  • 76,634
  • 23
  • 210
  • 236
3
votes
2 answers

Is it possible to catch die messages that happen inside another perl module?

I have a SOAP client written in Perl using LWP package for the HTTPS transport and XML::Simple for the parsing of the XML payloads. From time to time the call to XMLin fails with a die() and then my script dies and has to be restarted by a…
mathematician1975
  • 21,161
  • 6
  • 59
  • 101
3
votes
2 answers

Wide character in subroutine entry when using LWP with NTLM authentication

I'm trying to get LWP to work against an IIS server configured with NTLM authentication. When NTLM authentication is turned off on the server, the code work fine, so I assume that the only problem here is the NTLM authentication. So far, I have the…
Vetle
  • 3,287
  • 2
  • 27
  • 28
2
votes
1 answer

perl LWP: connection timeout different from request timeout

I'm using LWP::UserAgent to communicate with webservices on several servers; the servers are contacted one at a time. Each response might take up to 30 minutes to finish, so I set the LWP timeout to 30 minutes. Unfortunately the same timeout also…
spuelrich
  • 194
  • 7
2
votes
1 answer

decode form-urlencoded to hash

The response I get to an LWP request is application/x-www-form-urlencoded is it possible convert the text of this to a hash via some object method?
xenoterracide
  • 16,274
  • 24
  • 118
  • 243
2
votes
1 answer

How to scrape, using LWP and a regex, the date argument to a javascript function?

I'm having difficulty scraping dates from a specific web page because the date is apparently an argument passed to a javascript function. I have in the past written a few simple scrapers without any major issues so I didn't expect problems but I am…
SlowLearner
  • 7,907
  • 11
  • 49
  • 80
2
votes
1 answer

LWP::Simple getstore does not work in Windows

As a part of my Perl script I have the following simple command which is supposed to download a $url and store its contents to a file $file: getstore("$url", "$file"); This works perfectly fine when I run it on my fedora Linux, however when I run…
Eugene S
  • 6,709
  • 8
  • 57
  • 91
2
votes
1 answer

How do I crawl through a welcome page with Perl LWP?

I'm trying to crawl this page using Perl LWP: http://livingsocial.com/cities/86/deals/138811-hour-long-photo-session-cd-and-more I had code that used to be able to handle living social, but it seems to have stopped working. Basically the idea was to…
Vijay Boyapati
  • 7,632
  • 7
  • 31
  • 48
2
votes
1 answer

Perl LWP cannot connect to localhost on Windows 11

I'm writing a small Perl script to test an API in a local dev environment. I am using Strawberry Perl on Windows 11. The application is in ASP.NET MVC. When I run it, I get the following error: 500 Can't connect to localhost:44311 (Bad file…
thran
  • 137
  • 10
1 2 3
…
24 25