Questions tagged [lwp]

The World-Wide Web library for Perl - LWP is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web.

The main focus of the library is to provide classes and functions that allow you to write WWW clients. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

Most modules in this library provide an object oriented API. The user agent, requests sent and responses received from the WWW server are all represented by objects. This makes a simple and powerful interface to these services. The interface is easy to extend and customize for your own needs.

The main features of the library are:

  • Contains various reusable components (modules) that can be used separately or together.
  • Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources.
  • Provides a full object oriented interface or a very simple procedural interface.
  • Supports the basic and digest authorization schemes.
  • Supports transparent redirect handling.
  • Supports access through proxy servers.
  • Provides parser for robots.txt files and a framework for constructing robots.
  • Supports parsing of HTML forms. Implements HTTP content negotiation algorithm that can be used both in protocol modules and in server scripts (like CGI scripts).
  • Supports HTTP cookies.
  • Some simple command line clients, for instance lwp-request and lwp-download.

More information on the CPAN website.

365 questions
5
votes
2 answers

HTTPS Request with Perl UserAgent

I am trying to request an html document with HTTPS authentication using Perl. I haven't had an issue with non-HTTPS code in the past. I'm currently using LWP::UserAgent: #! /usr/bin/perl use LWP; use HTTP::Request::Common; my $browser =…
Soma Holiday
  • 185
  • 1
  • 1
  • 13
5
votes
3 answers

How do I use Perl's LWP to log in to a web application?

I would like to write a script to login to a web application and then move to other parts of the application: use HTTP::Request::Common qw(POST); use LWP::UserAgent; use Data::Dumper; $ua = LWP::UserAgent->new(keep_alive=>1); my $req = POST…
maxjackie
  • 22,386
  • 6
  • 29
  • 37
4
votes
1 answer

failed connect or "certificate verify failed" on LWP HTTPS GET

I posted this problem on Perl Monks yesterday but it worked for everyone who tried it (see http://www.perlmonks.org/?node_id=909968). However, I was using a different URL hoping to simplify the problem. I'm attempting to connect to api.betfair.com…
bohica
  • 5,932
  • 3
  • 23
  • 28
4
votes
1 answer

Accessing IPv6 resolvable URL with port (e.g. localhost:12345) results in Bad Address in Strawberry Perl 5.30.1

When using strawberry perl 5.30.1 under Windows 10 with IPv6 enabled, URLs with Portnumbers cannot be resolved properly due to what appears to be a bug in the DNS parser of Perl. For the following test, we have a simple webserver listening on Port…
DJGummikuh
  • 108
  • 6
4
votes
1 answer

Problem accessing Google API protected with OAuth2 by using LWP::Authen::OAuth2

I am currently coding a service on a perl server, that shall send a request to the Firebase Cloud Messaging API, which will then send push notifications to an app instance. Since FCM is part of the Google API family, an OAuth2 token is required to…
procra
  • 535
  • 4
  • 29
4
votes
1 answer

How can I handle proxy servers with LWP::Simple?

How can I add proxy support to this script? use LWP::Simple; $url = "http://stackoverflow.com"; $word = "how to ask"; $content = get $url; if($content =~ m/$word/) { print "Found $word"; }
localhost
  • 863
  • 6
  • 12
  • 12
4
votes
2 answers

Can I ask Perl 6's LWP::Simple to handle malformed UTF-8?

I'm using LWP::Simple to fetch a webpage that has a couple of malformed characaters in it. My call to .get blows up on that. Rather than that, I'd like to have the decode insert replacement characters in the confused parts and keep going. It looks…
brian d foy
  • 129,424
  • 31
  • 207
  • 592
4
votes
2 answers

How can I send cookies with Perl's LWP::Simple?

use LWP::Simple; use HTML::LinkExtor; use Data::Dumper; #my $url = shift @ARGV; my $content = get('example.com?GET=whateverIwant'); my $parser = HTML::LinkExtor->new(); #create LinkExtor object with no callbacks $parser->parse($content); #parse…
Neo
  • 11,078
  • 2
  • 68
  • 79
4
votes
1 answer

How may I bypass LWP's URL encoding for a GET request?

I'm talking to what seems to be a broken HTTP daemon and I need to make a GET request that includes a pipe | character in the URL. LWP::UserAgent escapes the pipe character before the request is sent. For example, a URL passed in…
bchgys
  • 53
  • 5
4
votes
1 answer

perl 500 certificate verify failed

I am using the WWW::Salesforce perl library to login and fetch some data. But I get this error on login - perl 500 certificate verify failed I've installed LWP::UserAgent Mozilla::CA and CACertOrg::CA but it didnt fix the issue. Is there any…
jacksparrow007
  • 1,298
  • 4
  • 19
  • 30
4
votes
2 answers

How can I get the body of an HTTP response using LWP::UserAgent in Perl?

I find that the return from LWP::UserAgent->request() contains both the header and body of a HTTP response. I just need the body of the response to do some parsing, so how can I do?
Haiyuan Zhang
  • 40,802
  • 41
  • 107
  • 134
4
votes
1 answer

How do I change the order of HTTP request headers sent by Perl's LWP?

For a test i need to do a get requets to a website - unfortunatly when using perl lwp the "connection" appears in the header b4 the host. As a result the request gets filtered by the web application firewall. All i need is to remove or move down the…
4
votes
3 answers

Is it possible to abort a HTTP request depending on the `Content-Type` when using Perl's LWP?

I want my script to download only text/html content and not binary or images that could take significantly more time to download. I know about the max_size parameter but I would like to add a check on the Content-Type header. Is this doable ?
mbonnin
  • 6,893
  • 3
  • 39
  • 55
4
votes
2 answers

No content from LWP request

SOLVED: I fixed it by installing HTML::HeadParser. But I still have no idea why it just suddenly stopped working. I noticed some LWP request that have worked before had stopped working so I made a small script to check why. For some reason I don't…
GraveRaven
  • 49
  • 4
3
votes
3 answers

Make timeout work for LWP::UserAgent HTTPS

Solution As reported by @limulus in the answer I accepted, this was a bug in Net::HTTPS version 6.00. Always be wary of fresh .0 releases. Here's the relevant diff between the buggy and fixed version of that module: D:\Opt\Perl512.32 :: diff…
Lumi
  • 14,775
  • 8
  • 59
  • 92