can't retrieve content of a webpage with perl's LWP agent

Question

I have a program that can retrieve the content of a specific webpage, but there are some pages where I get an error:

Can't get http://www.sitename.com
302 Moved Temporarily at geturl.pl line 30.

The site displays fine on a browser.

Wonder what I could do to get the content?

My code is very simple, the standard use of LWP and works fine on most pages.

  my $browser = LWP::UserAgent->new(
    agent=>'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)',
    keep_alive=>'1'
  );
  ...
  my $response = $browser->get($url);

Thanks!

================

update:

Yes this is the actual code I'm using. Is there an explicit option to turn on following redirects?
Yes wget works

Thanks

Is this the actual code you are using? The default configuration for `LWP::UserAgent` follows 302 redirects automatically. — friedo, Oct 20 '11 at 21:02
Does it work commandline, using wget or curl? A browser (which?) sometimes does more than just send the GET. — Konerak, Oct 20 '11 at 21:03

score 0 · Answer 1 · answered Jul 29 '13 at 19:56

I just read slides from a talk about some of various modules that can do HTTP in Perl; maybe you could try one of the others, like HTTP::Tiny:

perl -MHTTP::Tiny -E '$res=HTTP::Tiny->new->get("http://www.sitename.com/"); say join "\n", map { $res->{$_} } (qw(response status reason content))'

score 0 · Answer 2 · edited Oct 24 '11 at 16:03

The LWP::UserAgent docs indicate that the request method on the user agent will follow redirects automatically. It's unclear from this documentation if get uses the same logic.

You could use the request method by creating an HTTP::Request object. This example uses the request method:

perl -MData::Dumper -MHTTP::Request -MLWP -e '
  $request=HTTP::Request->new(GET => "http://www.google.com");
  $ua=LWP::UserAgent->new;
  print Dumper $ua->request($request);'

can't retrieve content of a webpage with perl's LWP agent

2 Answers2