1

I am using express and busboy to upload a file (from an Angular app).

If a large file is being uploaded and an error is detected - such as "file path does not exist", the only way to respond to the client is to wait for the full file upload stream to finish - even if nothing is happening with the file.

Using busboy, within the busboy.on('file', async (...) => { ... }) event function, if an error is encountered I use this code to reply to the client:

    const result = await doStuffWithFileStream(file).catch( error => {
        fileUploadErrors.push(error);
    });

    if (fileUploadErrors.length > 0) {
        res.json({ success: false, errors: fileUploadErrors });
    }

If the upload is quite large, I get the following (incorrect) client HttpErrorResponse:

{
  "headers": {
    "normalizedNames": {
    },
    "lazyUpdate": null,
    "headers": {
    }
  },
  "status": 0,
  "statusText": "Unknown Error",
  "url": "http:\/\/192.168.0.128:4500\/api\/project\/5d4c1e3375895a00310746dd\/upload\/qcgrnwut-igivxoqa-rqqphqhx",
  "ok": false,
  "name": "HttpErrorResponse",
  "message": "Http failure response for http:\/\/192.168.0.128:4500\/api\/project\/5d4c1e3375895a00310746dd\/upload\/qcgrnwut-igivxoqa-rqqphqhx: 0 Unknown Error",
  "error": {
    "isTrusted": true
  }
}

However, if I modify the busboy.on('file', async (...) => { ... }) function and prepend the following, to read through the file stream:

    file.on('data', (data) => {
        console.log(data);
    });

The file upload stream (despite not going anywhere) completes, and the following HttpResponse is sent to the client (after the upload completes):

{
  "headers": {
    "normalizedNames": {
    },
    "lazyUpdate": null
  },
  "status": 200,
  "statusText": "OK",
  "url": "http:\/\/192.168.0.128:4500\/api\/project\/5d4c1e3375895a00310746dd\/upload\/qcgrnwut-igivxoqa-rqqphqhx",
  "ok": true,
  "type": 4,
  "body": {
    "success": false,
    "errors": [
      {
        "file": "TestVideo.wmv",
        "message": "Failed to upload to \/home\/joe\/uploads\/does-not-exist: File path does not exist"
      }
    ]
  }
}

From testing, it seems to be that if the response is sent before the upload stream is finished, the client doesn't receive the response, only an error (probably caused by the cut-off stream).

If the data stream is read in some way, it allows the upload to complete, so the response is received by the client.

However, it's very impractical to wait for the full upload to complete to respond with the error that was caught very early on.

How can I end the upload stream, while still allowing a response to be sent? I'm not sure if this is a problem with busboy, express or the angular client.

Edit:

After testing my API with Postman, and seeing it get's the correct response, I now think this is partly an Angular HttpClient issue.

Although, even chrome's dev tools network tab isn't showing a reply from the request, so I believe there is still a better way of handling it on the server.

koushikmln
  • 648
  • 6
  • 23
Joe Wearing
  • 147
  • 2
  • 10
  • I've also tried to respond with header `res.header('Connection', 'close');` and call `busboy.removeListener('file', fileListener)` but the same problem exists. – Joe Wearing Aug 10 '19 at 13:30
  • I've now also tried moving the "doStuffWithFileStream()" intensive work to another thread using worker_threads, thinking I was blocking express from sending the response, but this didn't work either. I tested the API with Postman, and the response does get sent correctly. Which now means it must be a client issue (Angular's http client). – Joe Wearing Aug 12 '19 at 23:16

0 Answers0