How do I do a chunked transfer-encoding upload with WWW:Mechanize?

Question

I'm attempting to use a particular web service, and I can successfully perform the upload with the following command:

curl -X POST  --header "Transfer-Encoding: chunked" -d @Downloads/file.pdf https://some.webservice/upload

I get back a json response indicate success.

However, I'm unable to figure out how to do the same with WWW::Mechanize.

$mech->post("https://" . $server . "/upload", Content_Type => 'multipart/form-data', Content => [upID => $upid, name => $dlfile, userID => 0, userK => 0, file_0 => [$dlfile]]);

This receives a similar json response with a big fat error message in it.

Do I need to explicitly set the Transfer-Encoding header first? Is there some other trick to it? Google's not shedding much light on this, Perlmonks neither, and the documentation's a little obtuse.

The reason chunked is only used when callback is used is because the content length is otherwise known in advance, and there's no point to using chunked encoding if the content length is known. — ikegami, Mar 03 '17 at 00:17

score 4 · Answer 1 · answered Mar 02 '17 at 22:09

Do you really need WWW::Mechanize? It is a subclass of LWP::UserAgent with additional functionality that gives browser-like functionality like filling in and submitting forms, clicking links, a page history with a "back" operation etc. If you don't need all of that then you may as well use LWP::UserAgent directly

Either way, the post method is inherited unchanged from LWP::UserAgent, and it's fine to use it directly as you have done

The way to send a chunked POST is to set the Content to a reference to a subroutine. The subroutine must return the next chunk of data each time it is called, and finally ann empty string or undef when there is no more to send

Is the data supposed to be a JSON string?

It's easiest to write a factory subroutine that returns a closure, like this

sub make_callback {
    my ($data) = shift;
    sub { substr($data, 0, 512, "") }
}

Then you can call post like this

my $payload = to_json(...);

$mech->post(
    "https://$server/upload",
    Content_Type => 'multipart/form-data',
    Content      => make_callback($payload)
);

Please be aware that all of this is untested

I'm ok with using LWP directly, I just don't want to resort to system()ing the curl command, nor would I want to rewrite what's already a large script in python or something else. Thank for this, I'll get to tinkering with it. Trying to follow along with what I was seeing in dev tools, and I don't think I understood how this worked until you pointed it out. — John O, Mar 02 '17 at 22:20
Is there a way to do this with other fields besides the file contents? I can make the file upload, but it stumbles on several other missing values (as near as I can tell). — John O, Mar 06 '17 at 07:53

score 4 · Accepted Answer · edited Mar 03 '17 at 21:25

You can do it using HTTP::Request::StreamingUpload

my $starttime = time();
my $req = HTTP::Request::StreamingUpload->new(
    POST     => $url,
    path    => $file,
    headers => HTTP::Headers->new(
        'Transfer-Encoding' => 'chunked' 
    ),
);

my $gen = $req->content;
die unless ref($gen) eq "CODE";

my $total = 0;
$req->content(sub {
    my $chunk = &$gen();
    $total += length($chunk);

    print "\r$total / $size bytes ("
        . int($total/$size*100)
        . "%) sent, "
        . int($total/1000/(time()-$starttime+1))
        . " k / sec ";

    return $chunk;
});

my $resp = $ua->request($req);
print "\n";

unless ($resp->is_success) {
    die "Failed uploading the file: ", $resp->status_line;
}

my $con = $resp->content;
return $con;

Thank you, this solution seems to work correctly... json response has no errors now. I've still got a few things to figure out besides the (previous) error, but I think I can get the rest now. — John O, Mar 03 '17 at 18:13

How do I do a chunked transfer-encoding upload with WWW:Mechanize?

2 Answers2