Stack Overflow
Stack Overflow

Questions tagged [lwp]

The World-Wide Web library for Perl - LWP is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web.

The main focus of the library is to provide classes and functions that allow you to write WWW clients. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

Most modules in this library provide an object oriented API. The user agent, requests sent and responses received from the WWW server are all represented by objects. This makes a simple and powerful interface to these services. The interface is easy to extend and customize for your own needs.

The main features of the library are:

  • Contains various reusable components (modules) that can be used separately or together.
  • Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources.
  • Provides a full object oriented interface or a very simple procedural interface.
  • Supports the basic and digest authorization schemes.
  • Supports transparent redirect handling.
  • Supports access through proxy servers.
  • Provides parser for robots.txt files and a framework for constructing robots.
  • Supports parsing of HTML forms. Implements HTTP content negotiation algorithm that can be used both in protocol modules and in server scripts (like CGI scripts).
  • Supports HTTP cookies.
  • Some simple command line clients, for instance lwp-request and lwp-download.

More information on the CPAN website.

365 questions
7
votes
3 answers

handle lwp timeout effectively

I am using LWP to download content from web pages, and I would like to limit the amount of time it waits for a page. This is accomplished in lwp like this: my $ua = LWP::UserAgent->new; $ua->timeout(10); $ua->get($url); And this works fine, except…
srchulo
  • 5,143
  • 4
  • 43
  • 72
6
votes
2 answers

Perl LWP:UserAgent how to I add headers?

I'm a new perl programmer trying to convert a curl request into a Perl script get using LWP:UserAgent. The curl request example is: curl -X GET -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Cache-Control: no-cache" -H "Postman-Token:…
Talgarth
  • 69
  • 1
  • 1
  • 3
6
votes
2 answers

How can I get the ultimate URL without fetching the pages using Perl and LWP?

I'm doing some web scraping using Perl's LWP. I need to process a set of URLs, some of which may redirect (1 or more times). How can I get ultimate URL with all redirects resolved, using HEAD method?
planetp
  • 14,248
  • 20
  • 86
  • 160
6
votes
1 answer

I cannot Connect to any HTTPS site using LWP::UserAgent

I am trying to create a script that simply just connect to a website. However, for some reason it will not connect to anything that is using HTTPS. We have a proxy enabled here. However, I believe the proxy is not the problem, because if I were to…
theEli
  • 85
  • 5
6
votes
3 answers

LWP refuses to connect via HTTPS

I am running the following Perl snippet on Debian using Perl v5.14.2 and libwww-perl v6.04-1 my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new("GET", "https://google.com/"); my $rep = $ua->request($req); print $rep->status_line; This…
Tomas
  • 161
  • 1
  • 4
6
votes
3 answers

Compressing HTTP request with LWP, Apache, and mod_deflate

I have a client/server system that performs communication using XML transferred using HTTP requests and responses with the client using Perl's LWP and the server running Perl's CGI.pm through Apache. In addition the stream is encrypted using SSL…
user22410
  • 61
  • 1
  • 3
6
votes
8 answers

Automating sms sending by through Way2sms in Perl

I am trying to send sms through Way2sms using Perl LWP. The login part is being successful, after which I save the cookies to a local file. The welcome page after being logged in shows a Send SMS link, clicking on which one is redirected to another…
SexyBeast
  • 7,913
  • 28
  • 108
  • 196
6
votes
8 answers

HTTPS Proxy and LWP::UserAgent

I have read a number of threads on a number of sites and am still unable to make this work. I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an…
Ted B
  • 104
  • 1
  • 1
  • 6
5
votes
3 answers

How can a Perl web crawler follow an ASP.NET postback?

I'm building a webcrawler in Perl/LWP. How can the webcrawler follow a link in a ASP.NET grid like this: Next
prip
  • 91
  • 1
  • 4
5
votes
3 answers

Transparently Handling GZip Encoded content with WWW::Mechanize

I am using WWW::Mechanize and currently handling HTTP responses with the 'Content-Encoding: gzip' header in my code by first checking the response headers and then using IO::Uncompress::Gunzip to get the uncompressed content. However I would like…
Gurunandan Bhat
  • 3,544
  • 3
  • 31
  • 43
5
votes
1 answer

Windows / Perl / Net::SSLeay / OpenSSL: What locations are CA certificates loaded from?

Here's a program that does an HTTPS request, with some code at the start that I'm going to explain below: use 5.012; use LWP::UserAgent; use HTTP::Request::Common; use Net::SSLeay; BEGIN { return unless $^O eq 'MSWin32'; # only needed on…
Lumi
  • 14,775
  • 8
  • 59
  • 92
5
votes
2 answers

Perl LWP: why does IO::Socket::SSL use TLS 1.0, while Net::SSL uses TLS 1.2?

When I run the following code: use strict; use warnings; use IO::Socket::SSL; use LWP::UserAgent; my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0, }); my $res = $ua->get('https://internal.foo.bar.baz:20002'); print…
Matt Jacob
  • 6,503
  • 2
  • 24
  • 27
5
votes
1 answer

Using Different Outbound IPs in Perl

We have a few different websites running on the same server that all access 1 particular web service with each having their own unique API key. Unfortunately the web service has a daily limit based on IP address (not API key) so while each of our…
Russell C.
  • 1,649
  • 6
  • 33
  • 55
5
votes
1 answer

Inspecting SSL cert returned via LWP request

I'm requesting a web page using LWP in perl, and I'd like to be able to access the SSL certificate that the web server presents (I'm looking for an expiration date in the cert, among other things). The information I want isn't in the three headers…
dannysauer
  • 3,793
  • 1
  • 23
  • 30
5
votes
1 answer

PERL Get_Server_Certificate certificate verify failed error

I'm receiving a certificate error when trying to send a POST message to a website. The error I'm receiving: LWP::Protocol::https::Socket: SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed…
Rubix Rechvin
  • 571
  • 3
  • 16
1 2
3
…
24 25