Stack Overflow
Stack Overflow

Questions tagged [lwp]

The World-Wide Web library for Perl - LWP is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web.

The main focus of the library is to provide classes and functions that allow you to write WWW clients. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

Most modules in this library provide an object oriented API. The user agent, requests sent and responses received from the WWW server are all represented by objects. This makes a simple and powerful interface to these services. The interface is easy to extend and customize for your own needs.

The main features of the library are:

  • Contains various reusable components (modules) that can be used separately or together.
  • Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources.
  • Provides a full object oriented interface or a very simple procedural interface.
  • Supports the basic and digest authorization schemes.
  • Supports transparent redirect handling.
  • Supports access through proxy servers.
  • Provides parser for robots.txt files and a framework for constructing robots.
  • Supports parsing of HTML forms. Implements HTTP content negotiation algorithm that can be used both in protocol modules and in server scripts (like CGI scripts).
  • Supports HTTP cookies.
  • Some simple command line clients, for instance lwp-request and lwp-download.

More information on the CPAN website.

365 questions
10
votes
3 answers

Using LWP with SSL and client certificates

I'm porting an application from PHP/cURL to Perl and LWP::UserAgent. I need to do a POST request to a web server and provide a client certificate and key file. The PHP code I'm trying to replicate is this: curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,…
kent
  • 195
  • 1
  • 3
  • 11
9
votes
1 answer

LWP::UserAgent HTTP Basic Authentication

I tried to run this perl5 program: #!/usr/bin/env perl use strict; use warnings; …
XoR
  • 2,556
  • 4
  • 17
  • 15
9
votes
4 answers

Why don't my LWP::UserAgent credentials work?

I'm trying to access a protected file. Server is using digest authentication - which I can see from the printed out response. Here is the sample code: use LWP; use strict; my $url = 'http://somesite.com/aa/bb/cc.html'; my $username = 'scott'; my…
Ram
  • 3,034
  • 11
  • 38
  • 46
9
votes
2 answers

Perl WWW::Mechanize (or LWP) get redirect url

So I am using WWW::Mechanize to crawl sites. It works great, except if I request a url such as: http://www.levi.com/ I am redirected to: http://us.levi.com/home/index.jsp And for my script I need to know that this redirect took place and what the…
srchulo
  • 5,143
  • 4
  • 43
  • 72
8
votes
2 answers

How do I force LWP to use Crypt::SSLeay for HTTPS requests?

My symptom is that I cannot use a proxy with HTTPS requests with LWP. This seems to be a common problem, and the hints on Google and even here all suggest a work-around for setting the HTTPS_PROXY environment variable for use by Crypt::SSLeay. My…
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
8
votes
2 answers

lwp-request in shell: how to make POST request with body?

I use simple shell script to test some http server that process POST requests. Usually it looks like: echo "param1=value1¶m2=value2" | POST localhost/service But now I want to pass also some json in POST body and that's where I missing point…
Vlad Fedin
  • 498
  • 5
  • 8
8
votes
8 answers

How can I make LWP::UserAgent look like another browser?

This is my first post on SO, so be gentle. I'm not even sure if this belongs here, but here goes. I want to access some information on one of my personal accounts. The website is poorly written and requires me to manually input the date I want the…
SaulBack
  • 1,406
  • 2
  • 16
  • 31
7
votes
2 answers

Why does my image download with Perl's LWP give me the wrong-sized file?

I am trying to get an image from an HTTP server using Perl. I have the full URL of the file and am attempting to use my $data = LWP::Simple::get $params{URL}; my $filename = "image.jpg"; open (FH, ">$filename"); print FH $data; close (FH); Now,…
Xetius
  • 44,755
  • 24
  • 88
  • 123
7
votes
2 answers

WWW::Mechanize and Cookies

I use WWW::Mechanize::Shell to test stuff. Since I didn't managed to sign in on a web site I want to scrape, I thought I will use the browser cookie (chrome or firefox) for that specific website with the 'cookie' command WWW::Mechanize::Shell…
snoofkin
  • 8,725
  • 14
  • 49
  • 86
7
votes
1 answer

Perl equivalent of PHP's get_file_contents()?

The following PHP code does exactly what I want to do. The problem is I need to re-create it in Perl and I've been playing around with the open() and sysopen() Perl functions but can't do it. Does anyone have any help or know of any links that might…
dandemeyere
  • 341
  • 1
  • 4
  • 14
7
votes
2 answers

Who is wrong about the http://0:port?

The Plack suite commonly uses the http://0:port. E.g. the following plackup -MPlack::App::Directory -e 'Plack::App::Directory->new(root=>".");' prints HTTP::Server::PSGI: Accepting connections at http://0:5000/ However, the LWP::UserAgent (or some…
kobame
  • 5,766
  • 3
  • 31
  • 62
7
votes
1 answer

perl Client-SSL-Warning: Peer certificate not verified

I am having trouble with a perl screenscraper to an HTTPS site. In debugging, I ran the following: print $res->headers_as_string; and in the output, I have the following line: Client-SSL-Warning: Peer certificate not verified Is there a way I can…
Jeremey
  • 1,456
  • 3
  • 13
  • 19
7
votes
1 answer

LWP::UserAgent Can't Post with TLS1.1

Getting 500 handshaker error:443 over https. The host service I am sending XML to does not support TLS 1.2, they do support 1.0 and 1.1. Currently using LWP 6.03 on CentOS 6. Using the code below they claim I am still sending using TLS1.2 use…
chrisrth
  • 1,182
  • 3
  • 15
  • 36
7
votes
5 answers

PERL_LWP_SSL_VERIFY_HOSTNAME setting to 0 is not working

Am running into an issue connecting on a Ubuntu machine while my other machine works fine. The difference between both is the Ubuntu version and the SSLeay version but i can't narrow down what the issue is. I already did the following: a) add the…
user2601110
  • 71
  • 1
  • 1
  • 3
7
votes
1 answer

how does wget differ from Perl's lwp?

Im required to use wget for its ability to work like a crawler to develop one for my project. But all around in google searches im seeing people recommend to use LWP instead of Wget. Can you guys enlighten me on why this is so?
Marcus Lim
  • 567
  • 2
  • 5
  • 14
1
2
3
…
24 25