0

I have the following code:

#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $response = $ua->get('https://olms.dol-esa.gov/query/getYearlyData.do');
print $response->decoded_content;

The code takes upwards of two minutes to print out the results. Other web pages print out within a second or two. I'm curious to know why this page takes so long to load. How can I figure out what the problem is?

StevieD
  • 6,925
  • 2
  • 25
  • 45
  • 1
    This probably has nothing to do with LWP. Try visiting that page in a regular web browser and looking at the response times in the browser's developer tools. It's probably just a poorly-built page, or it's generating massive amounts of data. – ThisSuitIsBlackNot Feb 01 '17 at 22:45
  • For me, the page takes a few seconds to load in a browser, and a few seconds using your code above. Seems to be pretty slow on the server side, not sure there's much more to it. – jcaron Feb 01 '17 at 22:47
  • It loads in a couple of seconds in my browser but my script can take upwards of 2 minutes. Something else is at play. Also, I'm now noticiing about half the time I get an error: SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/local/share/perl/5.20.2/LWP/Protocol/http.pm line 48, line 1. – StevieD Feb 01 '17 at 23:19
  • Other web pages on the same site (https://olms.dol-esa.gov/) or other sites ? If other site, it may be a DNS problem. – Patrick Mevzek Feb 01 '17 at 23:59
  • Just this site seems to be giving me problems. – StevieD Feb 02 '17 at 01:09

1 Answers1

2

This hostname has serious DNS issues, see this DNSviz analysis. This means that "sometimes" you will have trouble resolving the hostname as an IP address. This may lead to timeouts.

Patrick Mevzek
  • 10,995
  • 16
  • 38
  • 54
  • Any idea how to at least make it fail fast if it can't resolve the name? – Chomeh Nov 25 '21 at 05:22
  • @Chomeh Use timeouts: https://metacpan.org/pod/LWP::UserAgent#timeout Or you can also do the name resolution yourself in advance of calling LWP::UserAgent. Also, why do you need to fail "immediately"? If you have high volume of things to check, doing them sequentially will always be a problem. – Patrick Mevzek Nov 25 '21 at 05:33
  • Cheers. The Oauth server became unavailable, but it took 10min for LWP to report 'Connection timed out' so I wasted a bunch of time checking other things. For some reason it's ignoring the configured timeout in my case. – Chomeh Nov 25 '21 at 06:46
  • A connection timeout is not a name resolution problem. – Patrick Mevzek Nov 25 '21 at 16:34