Perl LWP::UserAgent only reading first line of posted jpg file

Question

I'm sure this has been asked but I could not find a good question to my answer. I've got two scripts, one makes a post using LWP::UserAgent and the other basically receives the data, in this case I'm just looking to write a file. The file does get written but it can't be opened and the size is 1262 which leads me to believe only some of it is being read.

Here's what I'm doing (It's worth noting, I was uri/base64 encoding the file and passing everything via json, which was working, but my new task is the split the files out and pass everything as form params):

post script:

open (IMAGE, "./flower.jpg") or die "$!";
$raw_string1 = do{ local $/ = undef; <IMAGE>; };

my $req = HTTP::Request->new(POST => $url);
$req->content_type("application/x-www-form-urlencoded");
$req->content("json_string=$json&file_1=$raw_string1");

my $ua = LWP::UserAgent->new;
$res = $ua->request($req);
print $res->content;

Receiver script:

$cgi = CGI->new;
my $json_post = $cgi->param('json_string');
my $file_1 = $cgi->param('file_1');

open my $fh, '>', "$path/flower.jpg" or die $!;
binmode $fh;
print $fh $file_1;
close $fh;

Thanks for help in advance!

hooray for random down vote! I can provide more code, less code, better specifics about what I'm looking to do, but random down vote stinks. — kyle, Dec 03 '13 at 00:24
Thanks for response. I edited my above code to show in receiver script. — kyle, Dec 03 '13 at 00:34

ikegami · Accepted Answer · 2013-12-03T02:00:52.830

4

As previously mentioned, you have an encoding problem. The solution is simple:

my $req = HTTP::Request->new(POST => $url, [
   json_string => $json,
   file_1      => $raw_string1,
]);

which is short for

my $req = HTTP::Request->new(POST => $url,
   Content_Type => 'application/x-www-form-urlencoded',
   Content => [
      json_string => $json,
      file_1      => $raw_string1,
   ]
);

It's far more typical to use multipart/form-data to upload files, though. And if you do, you can even let HTTP::Request load the file for you!

my $req = HTTP::Request->new(POST => $url,
   Content_Type => 'multipart/form-data',
   Content => [
      json_string => $json,
      file_1      => [ 'flower.jpg' ],
   ]
);

CGI.pm will handle that no problem. See the section of the docs titled "PROCESSING A FILE UPLOAD FIELD".

edited Dec 03 '13 at 02:00

answered Dec 03 '13 at 01:33

ikegami

367,544
15
269
518

Thanks for the advice! I got it working using your advice. Many thanks! – kyle Dec 03 '13 at 02:45
I actually ended up going with this: my $url = ""; my $data = [json_string => $json, 'flower.jpg' => $raw_string1, 'fw4.pdf' => $raw_string2]; my $ua = LWP::UserAgent->new; $res = $ua->request(POST($url, $data)); print $res->content; – kyle Dec 04 '13 at 20:11
What a weird format. File names as identifiers?! With form-data, your fields can have both file names and file contents. Again, if you're designing a protocol, I recommend form-data. – ikegami Dec 04 '13 at 20:15
Well, I am not allowed to base64 encode anything and just stick it in my json arrays, which is what I was originally doing. So, I've been tasked with accepting a bunch of json arrays via a json param, then separately and as form params, a bunch of file name params. The last key value pair of the each json array is a file name so I can identify and stick the file somewhere when handling each array. In testing, I'm looking to be spinning through a few hundred of these. I'm certainly open to suggestions. Also, I did run into issues using your suggested syntax when trying to post to https. – kyle Dec 04 '13 at 20:31
Again, thanks for your help and your explanations allowed me to get a better grasp on things and get something working. *edit* probably also fair to note, I will not be writing the post side, just the receiver script. – kyle Dec 04 '13 at 20:31

score 1 · Answer 2 · answered Dec 03 '13 at 01:15

You're not encoding the $raw_string1 data before sticking it into the HTTP POST body. That means that if the data in flower.jpg includes a hex 0x26 byte (the & char) -- at position 1263, let's say -- then the POST data will look like this:

json_string={ ... JSON data ... }&file_1=...1262 bytes of raw JPEG data...&...more JPEG data...

... which means that anything parsing the form body for form variables will truncate file_1 after 1262 bytes.

I'd recommend continuing to encode the $raw_string1 data with base64 or something similar, even if you pass it through as its own POST variable rather than adding it to the JSON data.

Perl LWP::UserAgent only reading first line of posted jpg file

2 Answers2