Multiple file upload with mojolicious fails on large number of files

Question

I've hit a wall and my google skills have this time failed me. I'm in the process of learning mojolicious to create a useful front end for a series of Perl scripts that I frequently use. I've got a long way through it but I'm stumped at (multiple) file uploads when the total number of files reaches 950.

Previously, I encountered the problem where- in multiple file uploads- files would begin to be uploaded, but once the filesize reached 16 mb the upload stopped. I fixed this by setting $ENV{MOJO_MAX_MESSAGE_SIZE} = 50000000000. However, this problem is different. To illustrate, this is part of my script where I try to grab the uploaded files:

my $files = $self->req->every_upload('localfiles');
for my $file ( @{$files} ) {
    my $fileName = $file->filename =~ s/[^\w\d\.]+/_/gr;
    $file->move_to("temporary_uploads/$fileName");
    $self->app->log->debug("$fileName uploaded\n");
    push @fileNames, $fileName;
};
say "FILES: ".scalar(@fileNames);

I apologise that it may be ugly. If I attempt to upload 949 files, my array @fileNames is populated correctly, but if I try to upload 950 files, my array ends up empty, and it seems as though $files is empty also. If anyone has any ideas or pointers to guide me to the solution I would be extremely grateful!

Is there another server that wraps your Mojolicious app? That server may have a request size cap, which you may or may not be able to change. [Here's an example for nginx](https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/). — mob, Jun 15 '17 at 19:40
@mob Thanks for the hint there- it seems mojo can use a server called hypnotoad, although I'm not sure if that's been initialised in my case as other instances where I see people using it, it seems to have been declared specifically. I will explore this- thanks. — user1597452, Jun 16 '17 at 08:10
@mob my mistake- the server I'm using that wraps around it is called Morbo... that should have been obvious to me! — user1597452, Jun 16 '17 at 08:27
@SinanÜnür actually... uploading a single file of > 5gb went ok.... leaving me even more confused! — user1597452, Jun 16 '17 at 08:48
By way of update, I have now discovered that it is not a file size issue but an issue with the number of files- I can upload 949 1 byte files, but not 950 — user1597452, Jun 16 '17 at 09:16
Thanks for the pointer @SinanÜnür- it turned out that it wasn't actually to do with mojolicious- I had to increase the number of open file descriptors per process! I have resolved this- thanks for the pointers! — user1597452, Jun 16 '17 at 13:35

score 2 · Accepted Answer · answered Jun 16 '17 at 14:15

If I attempt to upload 949 files, my array @fileNames is populated correctly, but if I try to upload 950 files, my array ends up empty, and it seems as though $files is empty also.

That means the process is running out of file descriptors. In particular, the default for the Linux kernel is 1024:

For example, the kernel default for maximum number of file descriptors (ulimit -n) was 1024/1024 (soft, hard), and has been raised to 1024/4096 in Linux 2.6.39.

Multiple file upload with mojolicious fails on large number of files

1 Answers1