busboy - is there a way to send the response when all files have been uploaded?

Question

I'm trying to upload files to a server using node.js as backend and angular.js as frontend. I'm using express 4 + busboy for this. I have a table in the frontend where I should display all the files I'm uploading. So if I have 3 files and click on upload, angular should post these files to node.js and after getting the response back, refresh the table with those three files. This is the function I'm using in angular:

function uploadFiles(files){
    var fd = new FormData();
    for(var i = 0; i<files.length; i++){
        fd.append("file", files[i]);
    }
    $http.post('http://localhost:3000/upload', fd, {
        withCredentials: false,
        headers: {'Content-Type': undefined },
        transformRequest: angular.identity
    }).success(refreshTable()).error(function(){
        console.log("error uploading");
    });
}

and this is from node.js:

app.post('/upload', function(req, res) {
  var busboy = new Busboy({ headers: req.headers });
  busboy.on('file', function (fieldname, file, filename) {
    console.log("Uploading: " + filename);
    var fstream = fs.createWriteStream('./files/' + filename);
    file.pipe(fstream);
  });
  busboy.on('finish', function(){
    res.writeHead(200, { 'Connection': 'close' });
    res.end("");
  });
  return req.pipe(busboy);
});

the problem is that if I upload three files, as soon as the first file has been uploaded node.js sends the response and hence the table is updated only with the first file uploaded, if I refresh the page, the rest of the files appear. I think the problem is with this line in node: return req.pipe(busboy); if I remove that line, the post response keeps on pending for a long time and nothing happens, I think this is an async problem, anybody knows if there's a way to send the response back only when all files have been uploaded? thanks

score 10 · Accepted Answer · answered May 02 '15 at 00:11

A simple and common solution to this particular problem is to use a counter variable and listening for the finish event on the fs Writable stream. For example:

app.post('/upload', function(req, res) {
  var busboy = new Busboy({ headers: req.headers });
  var files = 0, finished = false;
  busboy.on('file', function (fieldname, file, filename) {
    console.log("Uploading: " + filename);
    ++files;
    var fstream = fs.createWriteStream('./files/' + filename);
    fstream.on('finish', function() {
      if (--files === 0 && finished) {
        res.writeHead(200, { 'Connection': 'close' });
        res.end("");
      }
    });
    file.pipe(fstream);
  });
  busboy.on('finish', function() {
    finished = true;
  });
  return req.pipe(busboy);
});

The reason for this is that busboy's finish event is emitted once the entire request has been fully processed, that includes files. However, there is some delay between when there is no more data to write to a particular file and when the OS/node has flushed its internal buffers to disk (and the closing of the file descriptor). Listening for the finish event for a fs Writable stream lets you know that the file descriptor has been closed and no more writes are going to occur.

thanks, I've tried that but still happening the same, this callback "refreshTable()" which is called after a successful post in angular (you can see it in the first code example) is called as soon as the first file has been uploaded, after that, the second file is being uploaded but the callback doesn't execute again, and if I move "return req.pipe(busboy);" into the finish event for a fs writable, angular keeps waiting for a post response and nothing happens. — nelson687, May 02 '15 at 19:06
could make it work, it turned out that I had something wrong in the angular code I posted, if you take a look I had "}).success(refreshTable())" and when I actually created an anonymous function in success and inside I called to refreshTable() it worked, didn't know that could be different, thanks! — nelson687, May 02 '15 at 19:31
@mscdex What If I've to upload the file to s3 & get the url link, instead of writing to the disk? any comments on that? — Ankit Balyan, Jan 04 '18 at 13:15

busboy - is there a way to send the response when all files have been uploaded?

1 Answers1