How to make a Mojolicious::Plugin::Minion job handle a large file upload

Question

I am trying to handle large file uploads with Mojolicious::Plugin::Minion so that I can respond to the user or the REST API caller quickly.

    # app start up   
     sub startup {
        ...
        $self->minion->add_task(upload_file => sub {
                                        my ($job, $file) = @_;
                                        my $filename = $file->filename;
        });
        ...
      }

    # Controller:
      sub upload_file {
            my $c = shift;
            $c->openapi->valid_input or return;

            # curl  -k -F name=test -F filepath=@/tmp/test.txt http://endpoint-to-uplod_file
            my $id = $c->minion->enqueue(upload_file => [$c->param('filepath')]);

            return $c->render(openapi => {
                message => q|request to upload file has been received Successfully.
                       This File will be zipped, encrypted and finally transferred to cloud.|,      
                statuscheck_endpoint => "uploadStatus/$id",
                jobid => $id
            });
       }

It looks like the minion doesn't get the object. I get the following error

Can't locate object method "filename" via package "Mojo::Upload=HASH(0x34451d0)" (perhaps you forgot to load "Mojo::Upload=HASH(0x34451d

Given that a minion job is run as child process, is this kind of IPC even possible?

Please help me to understand. I think that, until the file is uploaded, the server and the client are busy transferring file data and the server is unable to send back the response even though the minion can handle this.

Does it work for small file uploads? – mob Jan 19 '18 at 17:08 — mob, Jan 19 '18 at 17:08

Borodin · Accepted Answer · 2018-01-19T20:41:23.313

You can't pass objects (or any form of reference) as parameters in the enqueue call, as they would be meaningless in the context of the minion

This statement

my $id = $c->minion->enqueue( upload_file => [ $c->param('filepath') ] );

is passing the serialised version of the Mojo::Upload object, which is the string "Mojo::Upload=HASH(0x34451d0)", so it makes no sense to call the filename method on it

Having said that, I'm afraid I can't offer a way to make this work. The data from the file is being sent as part of an HTTP request, and it doesn't make sense to send a response to that request before the server has finished receiving it. Making the server asynchronous doesn't mean your client can do the same thing

There's not much that makes sense from the client's point of view: it still has to be running code to transfers the file's contents to the server. You can't just continue with an interactive session while the transfer continues unless you want to do the transfer outside the normal HTTP dialogue

Thank you. Yep that was my understanding as well. I wanted to verify my understaning. You made me clear. — Sachin Dangol, Jan 20 '18 at 00:07

How to make a Mojolicious::Plugin::Minion job handle a large file upload

1 Answers1