Loggin into website with LWP and Perl

Question

Somewhat inexperienced programmer here trying to write a program to log into my courses site and download all the content (lectures homeworks etc). Obviously it is a password protected site so I have to give it that. I understand LWP::UserAgent and the likes well enough, and that I need to use credentials. What I cannot figure out is how to get to the next page. I can go to the log-in, but how does perl get the result of my log-in?

code example (i pulled out the log-info obviously):

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $url = 'login URL';
$ua -> credentials(
  $url,
  '',
  'user',
  'pass'
);
my $response = $ua ->get($url);
print $response->content;

the content from response is the same content as what I would have got as if I had not passed any credentials. Obviously I'm missing something here....

Oh one other thing my own courses site does not have a unique url as far as I know.

score 5 · Accepted Answer · answered May 20 '11 at 15:43

5

You probably want to be using WWW::Mechanize, a subclass of LWP::UserAgent designed to act more like a browser, allowing you to navigate through pages of a website with cookie storage already taken care of for you.

answered May 20 '11 at 15:43

ysth

96,171
6
121
214

Thanks as well. Again some more reading! (I knew it was there, but sometimes I guess its all about looking in the right places) – msikd65 May 20 '11 at 16:06

score 2 · Answer 2 · answered May 20 '11 at 15:32

2

You only use credentials if the site uses HTTP basic auth, in which case you don't "log in", you just pass the credentials with every request.

If the site has a form based login system, then you need to use cookie_jar and request the form's action URI with whatever data it expects.

answered May 20 '11 at 15:32

Quentin

914,110
126
1,211
1,335

thanks for the fast reply. I guess I need to read up on HTTP and how the web works etc. – msikd65 May 20 '11 at 16:05

score 0 · Answer 3 · answered Jan 09 '15 at 11:43

#!/usr/bin/perl

use LWP::UserAgent;
use HTTP::Cookies;

my  $ua=LWP::UserAgent->new(timeout => 20);
    $ua->agent('Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.8) Gecko/20100202 MRA 5.5 (build 02842) Firefox/3.5.8');
    $ua->requests_redirectable(0);

my  $cook = HTTP::Cookies->new;
    $ua->cookie_jar($cook);

print = requester('http://urlexample/login.php', 'login=yourlogin&password=pass' )->as_string;

sub requester
{
    my $type = 'GET';
    if($_[1]){$type = 'POST'}   
    my $req = HTTP::Request->new($type => $_[0]);
    $req->content_type('application/x-www-form-urlencoded; charset=UTF-8');
    if($_[1]){$req->content($_[1])}
    my $res = $ua->request($req);
    return $res;
}

Loggin into website with LWP and Perl

3 Answers3