I'm using the following Perl code to get data from https://www.otcmarkets.com/research/stock-screener/api?sortField=symbol&sortOrder=asc&page=0&pageSize=20000:
use warnings;
use WWW::Mechanize::GZip;
my $TempFilename = "D:\\temp\\test.txt";
my $mech = WWW::Mechanize::GZip->new(
ssl_opts => {
verify_hostname => 0,
},
);
$mech->get("https://www.otcmarkets.com/research/stock-screener/api?sortField=symbol&sortOrder=asc&page=0&pageSize=20000");
open(OUT, ">", $TempFilename);
binmode(OUT, ":utf8");
print OUT $mech->content;
close(OUT);
Unfortunately the request always times out, and my temporary file always contains
read timeout at C:/Strawberry/perl/vendor/lib/Net/HTTP/Methods.pm line 268.
However, if I point a web browser to the same URL, I get a bunch of JSON data that looks like this, which is what I am seeking:
"{\"count\":17114,\"pages\":1,\"stocks\":[{\"securityId\":194057,\"reportDate\":\"Jan 26, 2022 12:00:00 AM\",\"symbol\":\"AAAIF\",\"securityName\":\"ALTERNATIVE INVESTMENT TR\",\"market\":\"Pink ...
My question is whether there is any way I can modify my script so that it saves the same data that my web browser is able to display instead of the timeout message to my file.
Thanks