0

I'm trying to retrieve a page using LWP::UserAgent but I keep getting a "500 Internal Server Error" as a response. Retrieving the exact same page in Firefox (using a fresh "Private Window" - so without any cookies set yet) succeeds without a problem.

I've duplicated the headers exactly as sent by Firefox, but that still does not make a difference. Here's my full code:

use strict;
use LWP::UserAgent;

my $browserObj = LWP::UserAgent->new();
$browserObj->cookie_jar( {} );
$browserObj->timeout(600);

my @header = (
  'Host' => 'www.somedomain.com',
  'User-Agent' => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0',
  'Accept-Language' => 'en-US,en;q=0.5',
  'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Encoding' => 'gzip, deflate, br',
  'DNT' => '1',
  'Connection' => 'keep-alive',
  'Upgrade-Insecure-Requests' => '1'
);

my $URL = "https://www.somedomain.com";

my $response = $browserObj->get( $URL, @header );
if( $response->is_success ) {
  print "Success!\n";
} else {
  print "Error: " . $response->status_line . ".\n" );
}

The real web address is something other than "www.somedomain.com". In fact, it's a URL to an online casino, but I don't want my question be regarded as spam.

But anyone any idea what could be wrong?

Zippy1970
  • 601
  • 6
  • 20
  • 3
    That's impossible to say without the real URL. The headers look fine. In any case, the server shouldn't respond with 500 if you are doing something wrong. 500 means the server did something wrong. Since you don't control the server, there seems to be not much we can do about it, unless the server is too stupid to behave correctly (which might be on purpose). – simbabque Jun 21 '17 at 20:19
  • 3
    Try to run your code with `perl -MIO::Socket::SSL=debug4 program.pl` which gives you information in case the problem is at the SSL level. Also add the full response you got since the headers usually contain a more detailed error description. And please add the version of LWP you are using. And if you are using a proxy make sure to use at least LWP version 6.06. Apart from that: I'm pretty sure that LWP cannot deal with `br` compression but you announce this in `Accept-Encoding`. – Steffen Ullrich Jun 21 '17 at 20:43
  • 1
    LWP::UserAgent will also return a 500 error on a connection failure, making the presumption that the server is at fault but this is by no means guaranteed. Are you behind a proxy? – Nick P Jun 22 '17 at 05:09
  • I propose to close the question since the OP is not providing any of the suggested information and shows no reaction to any of the comments. In the current state of missing information one can only guess what might be the case and since there are many possible reasons this makes the question too broad. – Steffen Ullrich Jun 23 '17 at 04:49

1 Answers1

1

On our corporate network which has a proxy (and an out of date perl version - there may be better options in newer versions) we tend to add the following for one-offs:

BEGIN {
    $ENV{HTTPS_DEBUG} = 1; # optional but can help if you get a response
    $ENV{HTTPS_PROXY} = 'https://proxy.server.here.net:8080';
}

If we don't do this the script simply fails to connect with no other information.

You may also want to add something like this if you want to inspect the messages:

$browserObj->add_handler("request_send",  sub { shift->dump; return });
$browserObj->add_handler("response_done", sub { shift->dump; return });
Nick P
  • 759
  • 5
  • 20
  • 1
    You can also use https://metacpan.org/pod/LWP::ConsoleLogger::Everywhere. Just drop it in and it will tell you all about all your LWP::UAs everywhere in your code. – simbabque Jun 22 '17 at 09:39
  • Note that installing as `cpan -i LWP::ConsoleLogger::Everywhere` results in an extensive rebuild, apparently attaching the *logger* code to every installed Perl module that uses LWP. Just saying. – CODE-REaD Sep 01 '18 at 15:38