Stack Overflow
Stack Overflow

Questions tagged [lwp]

The World-Wide Web library for Perl - LWP is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web.

The main focus of the library is to provide classes and functions that allow you to write WWW clients. The library also contain modules that are of more general use and even classes that help you implement simple HTTP servers.

Most modules in this library provide an object oriented API. The user agent, requests sent and responses received from the WWW server are all represented by objects. This makes a simple and powerful interface to these services. The interface is easy to extend and customize for your own needs.

The main features of the library are:

  • Contains various reusable components (modules) that can be used separately or together.
  • Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources.
  • Provides a full object oriented interface or a very simple procedural interface.
  • Supports the basic and digest authorization schemes.
  • Supports transparent redirect handling.
  • Supports access through proxy servers.
  • Provides parser for robots.txt files and a framework for constructing robots.
  • Supports parsing of HTML forms. Implements HTTP content negotiation algorithm that can be used both in protocol modules and in server scripts (like CGI scripts).
  • Supports HTTP cookies.
  • Some simple command line clients, for instance lwp-request and lwp-download.

More information on the CPAN website.

365 questions
3
votes
3 answers

How to check if downloaded file using getstore() function is not complete/corrupted?

I wrote a quick script to download files using LWP::Simple library and its getstore() function. It is working rather well, but occasionally downloaded file is not complete. I do not know what is causing this, but when I download it afterward…
Gargauth
  • 2,495
  • 6
  • 27
  • 30
3
votes
2 answers

LWP::useragent keep_alive not working

I am using below code to post JSON data using LWP::useragent. I want to keep my session open and post two requests but it seems that its not working on linux machine (two POST requests are sent in two sessions instead of one). Any suggestions?…
user2896215
  • 507
  • 7
  • 20
3
votes
2 answers

Waiting for CloudFlare DDOS Protection LWP Perl

Edit: Ended up using WWW::Mechanize::Firefox. I answered my own question below. I am trying to access a website and download it's page. The cloudflare DDOS protection on the site will occasionally come on, and I can't make LWP go through it. I can…
Spenser Truex
  • 159
  • 1
  • 11
3
votes
1 answer

LWP::UserAgent and login credentials

I'm trying to set up the credentials with LWP::UserAgent, but I'm not able to log in. $username, $passwd are correct. I don't understand what should I put in 3rd argument(according to dos $realm, here Authentication). Here the snippet: my $browser =…
user897237
  • 613
  • 5
  • 12
  • 25
3
votes
4 answers

Unable to install LWP::Simple on cygwin

I am a Perl beginner trying to install LWP::Simple on cygwin (CYGWIN_NT-6.3) but it fails to install. The installation seems to go wrong from the very start. Could you please help me to fix the problem? $ perl -MCPAN -e'install…
krenkz
  • 484
  • 6
  • 15
3
votes
1 answer

Perl LWP GET or POST to an SNI SSL URL

I have a system that sends data to customers using perl LWP. They can choose their URL and whether to POST or GET. A new customer recently complained that the service doesn't work and they suspect it's because their endpoint uses SNI SSL. Looking in…
Ben Holness
  • 2,457
  • 3
  • 28
  • 49
3
votes
4 answers

Why does Perl's LWP gives me a different encoding than the original website?

Lets say i have this code: use strict; use LWP qw ( get ); my $content = get ( "http://www.msn.co.il" ); print STDERR $content; The error log shows something like "\xd7\x9c\xd7\x94\xd7\x93\xd7\xa4\xd7\xa1\xd7\x94" which i'm guessing it's utf-16…
Ricky Levi
  • 7,298
  • 1
  • 57
  • 65
3
votes
4 answers

Why doesn't my Perl program output anything to my file?

I'm trying to write in file ~/.log but file is remaining empty. I don't know why this is happening, everything seems fine. My system is Ubuntu 9.10amd64, Perl 5.10. #!/usr/bin/perl # vim:set filetype=perl: use strict; use warnings; use LWP::Simple…
XoR
  • 2,556
  • 4
  • 17
  • 15
3
votes
4 answers

How can I extract XML of a website and save in a file using Perl's LWP?

How can I extract information from a website (http://tv.yahoo.com/listings) and then create an XML file out of it? I want to save it so to parse later and display information using JavaScript? I am quite new to Perl and I have no idea about how to…
shuwee
3
votes
2 answers

Suppressing "Day too big" warning in Perl LWP::UserAgent

I have a fairly simple perl script with uses the LWP::UserAgent module to follow URLs through redirection to find the final destination URL which it then stores in our MySQL database. The problem is that from time to time the script reports…
Russell C.
  • 1,649
  • 6
  • 33
  • 55
3
votes
2 answers

Perl Error: X-Died: Missing newline after chunk data:

While trying to get a webpage using LWP::UserAgent I am getting the following error X-Died: Missing newline after chunk data: 'OeXk0Yk1iRVhX....pOnR5cGU9InhzOnN' at /usr/share/perl5/Net/HTTP/Methods.pm line 481. To make things worse, it…
Jean
  • 21,665
  • 24
  • 69
  • 119
3
votes
2 answers

making LWP Useragent faster

I need to perform a large number of HTTP post requests, and ignore the response. I am currently doing this using LWP::UserAgent. It seems to run somewhat slow though I am not sure if it is waiting for a response or what, is there anyway to speed it…
stbtra
  • 31
  • 1
  • 3
3
votes
1 answer

Perl: Using HTTP::Request::Common to post with authentication

I tried to use HTTP::Request::Common to do a post to a URL. Unfortunately, it doesn't seem to work with authentication, and I had to switch back to plain ol' HTTP::Request. If I didn't need the authentication, I could have done the whole thing this…
David W.
  • 105,218
  • 39
  • 216
  • 337
3
votes
3 answers

How can I use Test::LWP::UserAgent if I cannot replace the $ua in the app code directly?

I've got a sub that retrieves some data from an API via a REST service. The code is rather simple, but I need to post parameters to the API and I need to use SSL, so I have to go through LWP::UserAgent and cannot use LWP::Simple. This is a…
simbabque
  • 53,749
  • 8
  • 73
  • 136
3
votes
2 answers

Why can't LWP::UserAgent get this site entirely?

It outputs only a few lines from the beginning. #!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new; my $response =…
user2048750
1 2 3
…
24 25