Downloading text files: Perl

Question

The files are not downloading, please help.

#!/usr/bin/perl -w
require HTTP::Response;
require LWP::UserAgent;
open (INPUT, "ndb_id_file.txt") or die "can't open ndb_id_file.txt";
@input = <INPUT>
foreach $line(@input) {
    $ua = LWP::UserAgent->new;
    $ua->env_proxy('http');
    $ua->proxy(['http', 'ftp'],'http://144020019:*******@netmon.****.ac.in:80');
    response = 
        $ua->get('www.ndbserver.rutgers.edu/files/ftp/NDB/coordinates/na-biol/$line');
    if ($response->is_success) {
        $content = $response->content();
        open(OUT,">$line.pdb") or die "Output file $line cannot be produced... Error...";
        print (OUT "$content");
    }
}

I used strict and warnings. It says that $response, content and line require an explicit package name. I checked whether perl modules are present by cpan LWP::UserAgent and cpan HTTP::Response through command line.It says both modules are upto date — phani, Apr 04 '15 at 16:35
That's funny. Even without those measures in place, I get `Unquoted string "response" may clash with future reserved word at E:\Perl\source\resp.pl line 10.`, `syntax error at E:\Perl\source\resp.pl line 6, near "$line("`, `syntax error at E:\Perl\source\resp.pl line 17, near "}"`, `Execution of E:\Perl\source\resp.pl aborted due to compilation errors.` Are you really unable to fix those problems yourself? ` — Borodin, Apr 04 '15 at 22:25

score 2 · Answer 1 · answered Apr 04 '15 at 22:21

There are a number of problems with your program. The major ones being in this line

response = $ua->get('www.ndbserver.rutgers.edu/files/ftp/NDB/coordinates/na-biol/$line');

You are trying to assign to response, which is not a variable name
the value of $line is not being inserted into the URL because you are using single quotes
The contents of $line end with a linefeed, which should be removed using chomp
The URL has no scheme — it should start with http://

Apart from those points, you should fix these issues

You must always use strict and use warnings at the top of every Perl program you write. Adding -w on the shebang line is far inferior
You should use rather than require LWP::UserAgent. And there is no need to also use HTTP::Response as it is loaded as part of LWP
You should always use the three-parameter form of open with lexical file handles. And if the open fails you should print a die string that includes the value of $! which gives the reason for the failure
You should use while to read data from a file one line at a time, unless you have a good reason to need all of it in memory at once
There is no need to create a new user agent $ua every time around the loop. Just make one and use it to fetch every URL
You should use decoded_content instead of content to fetch the content of an HTTP::Response message in case it is compressed

Here is a program that includes all of those fixes. I haven't been able to test it but it does compile

#!/usr/bin/perl
use strict;
use warnings;

use LWP::UserAgent;

my $in_file = 'ndb_id_file.txt';

open my $fh, '<', $in_file or die qq{Unable to open "$in_file" for input: $!};

my $ua = LWP::UserAgent->new;
$ua->env_proxy('http');
$ua->proxy(['http', 'ftp'], 'http://144020019:*******@netmon.****.ac.in:80');

while ( my $line = <$fh> ) {

    chomp $line;
    my $url = "http://www.ndbserver.rutgers.edu/files/ftp/NDB/coordinates/na-biol/$line";
    my $response = $ua->get($url);

    unless ( $response->is_success ) {
      warn $response->status_line;
      next;
    }

    my $content = $response->decoded_content;
    my $out_file = "$line.pdb";

    open my $out_fh, '>', $out_file or die qq{Unable to open "$out_file" for output: $!};
    print $out_fh $content;
    close $out_fh or die qq{Unable to close "$out_file": $!};
}

Downloading text files: Perl

1 Answers1