Node.js Streaming/Piping Error Handling (Change Response Status on Error)

Question

I have millions of rows in my Cassandra db that I want to stream to the client in a zip file (don't want a potentially huge zip file in memory). I am using the stream() function from the Cassandra-Node driver, piping to a Transformer which extracts the one field from each row that I care about and appends a newline, and pipes to archive which pipes to the Express Response object. This seems to work fine but I can't figure out how to properly handle errors during streaming. I have to set the appropriate headers/status before streaming for the client, but if there is an error during the streaming, on the dbStream for example, I want to clean up all of the pipes and reset the response status to be something like 404. But If I try to reset the status after the headers are set and the streaming starts, I get Can't set headers after they are sent. I've looked all over and can't find how to properly handle errors in Node when piping/streaming to the Response object. How can the client tell if valid data was actually streamed if I can't send a proper response code on error? Can anyone help?

function streamNamesToWriteStream(query, res, options) {
  return new Promise((resolve, reject) => {

    let success = true;

    const dbStream = db.client.stream(query);
    const rowTransformer = new Transform({
      objectMode: true,
      transform(row, encoding, callback) {
        try {
          const vote = row.name + '\n';
          callback(null, vote);
        } catch (err) {
          callback(null, err.message  + '\n');
        }
      }
    });

    // Handle res events
    res.on('error', (err) => {
      logger.error(`res ${res} error`);
      return reject(err);
    });

    dbStream.on('error', function(err) {
      res.status(404).send() // Can't set headers after they are sent.
      logger.debug(`dbStream error: ${err}`);
      success = false;
      //res.end();
      //return reject(err);
    });

    res.writeHead(200, {
      'Content-Type': 'application/zip',
      'Content-disposition': 'attachment; filename=myFile.zip'
    });

    const archive = archiver.create('zip');
    archive.on('error', function(err) { throw err; });
    archive.on('end', function(err) {
      logger.debug(`Archive done`);
      //res.status(404).end()
    });

    archive.pipe(res, {
      //end:false
    });
    archive.append(dbStream.pipe(rowTransformer), { name: 'file1.txt' });
    archive.append(dbStream.pipe(rowTransformer), { name: 'file1.txt' });
    archive.finalize();
  });
}

This is an interesting problem. Corrupted downloads are somewhat inherent to the way HTTP/TCP work (therefore the popularity of making hashes to available to validate after downloading). Looking forward for others' suggestions. — Víctor López García, Apr 04 '17 at 21:24

score 4 · Accepted Answer · answered Apr 04 '17 at 23:06

Obviously it's too late to change the headers, so there's going to have to be application logic to detect a problem. Here's some ideas I have:

Write an unambiguous sentinel of some kind at the end of the stream when an error occurs. The consumer of the zip file will then need to look for that value to check for a problem.
Perhaps more simply, have the consumer execute a verification on the integrity of the zip archive. Presumably if the stream fails the zip will be corrupted.

Node.js Streaming/Piping Error Handling (Change Response Status on Error)

1 Answers1