LWP GET large file download

Question

I have a special case for file download. I need to do chunked download for large files and also need to pass parameters to the CGI script prior to download.

It is really a REST interface. I have searched all over the Internet, and there are lots of stuff on the download part, and lots of stuff on the parameter part, but when I go to put them together I get errors. Also note that I do a POST in a similar way, and it works fine. Here is my code snip:

# $filename, $target, $url, $bs, etc. are all set...
my $bytes_received = 0;

open (FH, ">", "$filename") or $logger->error("Couldn't open $filename for writing: $!" );
my $ua = LWP::UserAgent->new();
my $res = $ua->get(
    $url,
    ':content_cb' => \&callback,
    'Content' => {
    "api" => 'olfs',
    "cmd" => 'rfile',
    "target" => $target,
    "bs" => $bs});

    print $logger->info("$bytes_received bytes received");

sub callback{
    my($chunk, $res) = @_;
    $bytes_received += length($chunk);
    print FH $chunk;
}

Here are the errors:

Not a SCALAR reference at /usr/local/share/perl5/HTTP/Message.pm line 163.
 at /usr/local/share/perl5/HTTP/Message.pm line 163
    HTTP::Message::add_content('HTTP::Request=HASH(0x1956a88)', 'HASH(0x7fdfda565e88)') called at /usr/local/share/perl5/HTTP/Request/Common.pm line 111
    HTTP::Request::Common::_simple_req(undef, undef) called at /usr/local/share/perl5/HTTP/Request/Common.pm line 20
    HTTP::Request::Common::GET('http://10.0.0.15:8084/cgi-bin/olss.cgi', 'Content', 'HASH(0x7fdfda565e88)') called at /usr/local/share/perl5/LWP/UserAgent.pm line 410
    LWP::UserAgent::get('LWP::UserAgent=HASH(0x191a220)', 'http://10.0.0.15:8084/cgi-bin/olss.cgi', ':content_cb', 'CODE(0x1845818)', 'Content', 'HASH(0x7fdfda565e88)') called at ./olfs_get.pl line 72
Debugged program terminated.  Use q to quit or R to restart,
  use o inhibit_exit to avoid stopping after program termination,
  h q, h R or h o to get additional info.
  DB<3> print oct("764")
500
  DB<4>

ikegami · Accepted Answer · 2013-10-31T00:39:27.677

1

$ua->get( $url )
$ua->get( $url , $field_name => $value, ... )

This method will dispatch a GET request on the given $url. Further arguments can be given to initialize the headers of the request.

There's no such thing as a Content header. ->post uses this to create the message-body, which is never used for GET requests. If you want to build a url, you can use URI.

$ua->post( $url, $field_name => $value,... Content => \%form )
$ua->post( $url, $field_name => $value,... Content => \@form )
$ua->post( $url, $field_name => $value,... Content => $content )

edited Oct 31 '13 at 00:39

answered Oct 31 '13 at 00:26

ikegami

367,544
15
269
518

Interesting, so I am using 'Content' when I POST and my server cgi uses $param->{'field'} to access the parameters that are part of this content. When I GET and try and add 'Content' there is not such header, so I get the errors. So I wonder how to access $field_name key value pairs in my CGI when I place a GET request? I suppose I should do a test and dump everything that comes in. – tradetree Oct 31 '13 at 04:24
I did a test on the server CGI side, and could not find any of my key / value pairs in the dump. I am doing "my $query = new CGI; my $rcvd_data = Dumper($query); print $rcvd_data, and I get: $VAR1 = bless( { '.parameters' => [], 'use_tempfile' => 1, '.charset' => 'ISO-8859-1', '.fieldnames' => {}, 'param' => {}, '.header_printed' => 1, 'escape' => 1 }, 'CGI' ); – tradetree Oct 31 '13 at 14:09
Is that a question? Where's the rest of it? (code, desired behaviour, etc)? – ikegami Oct 31 '13 at 15:59
Yes, it is a question. Should I start another thread? All the information is there. The question is, "Where are the key value pairs? They do not show up in the Dumper output." I wanted to mark this answer (your answer) as correct, but I can't find the field values in the CGI server end of things, so I don't know. – tradetree Oct 31 '13 at 16:38
What key value pairs? The information is not there. Like I said, where's your code? What's the desired behaviour or output? – ikegami Oct 31 '13 at 16:41
Ok, I will post a new question so I can show the code properly formatted and commented. – tradetree Oct 31 '13 at 17:50
New question posted here: http://stackoverflow.com/questions/19713846/lwp-get-large-file-download-headers-missing – tradetree Oct 31 '13 at 20:52

LWP GET large file download

1 Answers1