Is there a way to limit the max number of parallel processes that upload chunks in dropzone.js?

Question

We need to be able to support large uploads to our site. That's why we have implemented chunked uploading with Dropzone.js.

It works great, but when I use a very large file, I can see in Chrome's network debug view that all requests are immediatly started in a pending state and the browser isn't able to keep up. After some time, there are too many open requests and Chrome starts to return "net::ERR_INSUFFICIENT_RESOURCES" for some of the pending requests.

These are the relevant options of our Dropzone config:

[...]
  parallelUploads: 1,  // only one file is uploaded at a time
  maxFilesize: 8148, // max individual file size 8 GB
  chunking: true,      // enable chunking
  forceChunking: true, // forces chunking when file.size < chunkSize
  parallelChunkUploads: true, // allows chunks to be uploaded in parallel
  chunkSize: 2*1024*1024,  // chunk size 2MB
  retryChunks: true,   // retry chunks on failure
  retryChunksLimit: 3
[...]

Here is one of hundreds of insufficient_resources errors from the chrome console when uploading large files:

dropzone.js:9622 POST http://localhost:44802/AuthenticatedFolder/65fc862e-4cb1-41b0-a39c-f6a41ea5e64d/DoChunkedUpload/A8BB25EA676977DC6F064AEEE636E388FCF5E513 net::ERR_INSUFFICIENT_RESOURCES
submitRequest @ dropzone.js:9622
_uploadData @ dropzone.js:9406
handleNextChunk @ dropzone.js:9218
(anonymous) @ dropzone.js:9248
(anonymous) @ dropzone.js:9423
transformFile @ dropzone.js:7545
_loop @ dropzone.js:9419
_transformFiles @ dropzone.js:9429
uploadFiles @ dropzone.js:9171
processFiles @ dropzone.js:9078
processFile @ dropzone.js:9051
processQueue @ dropzone.js:9042
(anonymous) @ dropzone.js:8743
setTimeout (async)
enqueueFile @ dropzone.js:8742
(anonymous) @ dropzone.js:8705
accept @ dropzone.js:7407
accept @ dropzone.js:8669
addFile @ dropzone.js:8695
(anonymous) @ dropzone.js:8571
_addFilesFromItems @ dropzone.js:8595
drop @ dropzone.js:8510
drop @ dropzone.js:8225

I'm using the latest version 5 release of dropzone(5.9.3).

Is there a way to limit the number of parallel running requests? If not, it would be a great addition to the options. As a workaround I've disabled parallelChunkUploads. It works, but of course it also slows down the upload.

Thanks for any help!

score 1 · Answer 1 · answered May 30 '22 at 09:36

We have the exact same problem in an application at work. There's a merge request here https://gitlab.com/meno/dropzone/-/merge_requests/53 that provides a solution. I've asked about the status on it but the merge request is pretty old by now. The solution seems to be pretty easily implemented however, and our current plan is to copy the Dropzone source code and just do it ourselves. This is obviously not ideal since we'd be relying on a local copy of whatever version we copy, but it might be worth it until Dropzone decided to solve it themselves. I'll keep you posted

score 0 · Answer 2 · answered May 30 '22 at 13:17

A workaround that worked for my case is:

// Wrapper for upload process to prevent "Not enough resource error"
const oldFunction = fileUploader.submitRequest;
let inProgress = 0;
fileUploader.submitRequest = function(xhr, formData, files){
    runQueue(oldFunction,this,xhr, formData, files)
}
function runQueue(uploadDataFunction,scope,xhr, formData, files){
    if(inProgress <= 50){
        inProgress++;
        xhr.onreadystatechange = function () {
            if (xhr.readyState === 4) {
                inProgress--;
            }
        }
        uploadDataFunction.call(scope,xhr, formData, files);
    }else{
        setTimeout(function(){runQueue(uploadDataFunction,scope,xhr, formData, files)},100);
    }
}

I override the function that is responsible for processing the individual requests. Queue 50 processes in parallel (as far as I know Chrome processes a maximum of 2 processes in parallel) so that there is a buffer and all other processes are rechecked after a timeout and if there are less than 50 processes in queue they are added.

score 0 · Answer 3 · answered Apr 01 '23 at 06:04

One way to reduce the number of requests is to increase the chunkSize. You have it at 2M which means a 10G file is splitting into thousands of chunks (and upload requests). Depending on your server settings you can probably set chunkSize to around 1G or more, so your 10G file upload will only be split into about 10 chunks/processes.

Is there a way to limit the max number of parallel processes that upload chunks in dropzone.js?

3 Answers3