0

I'm querying a SQL server from an express api using the node-mssql module, and piping the rows (with some intermediate steps) to the response stream. I want to avoid writing headers until I know that the query won't produce an error before any rows, so that I can send a useful error message to the client instead of terminating the stream with the express default handler. However, express is writing default headers at some point that I'm having trouble identifying.

I've tried writing headers in a .once('data',) listener in the pipeline but at that point some default behavior has written headers for me and this produces an error. 

let request = await new sql.Request(pool);

res.writeHead(200, {
        'Transfer-Encoding': 'chunked',
        'charset' : 'utf-8',
        'Content-Type': 'application/json',
        'Content-Encoding': 'gzip'        
    })

ndjsonStream.on('error', next)

request.pipe(ndjsonStream)
    .pipe(transformer)
    .pipe(gzip)
    .pipe(res);

The above works but doesn't let me pass a useful error message.

let request = await new sql.Request(pool);

ndjsonStream.once('data', () => {
    res.writeHead(200, {
        'Transfer-Encoding': 'chunked',
        'charset' : 'utf-8',
        'Content-Type': 'application/json',
        'Content-Encoding': 'gzip'        
    })
})

ndjsonStream.on('error', next)

request.pipe(ndjsonStream)
    .pipe(transformer)
    .pipe(gzip)
    .pipe(res);

request.query(query);

This produces an error because express has helpfully written some headers for me before this listener fires i.e. content-type: text/html.

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
Surplus Yogurt
  • 43
  • 1
  • 6
  • 1
    Unless you buffer everything into memory you cannot ensure an error will not occur after the response has started streaming to the client. An error can occur at any time during streaming. The headers have to be sent before the response body so express likely sends some default headers when response data starts to be written. You can try using `res.setHeader` to force it to use a certain content type, but error detection needs to occur before you start piping data to the response. – Jake Holzinger Jul 11 '19 at 00:28
  • Check here https://stackoverflow.com/questions/31844316/node-js-check-if-stream-has-error-before-piping-response – Crusader Jun 10 '20 at 10:48

1 Answers1

1

The way an http response works, you HAVE to write the http headers before you start sending any data. That's just how the http response format was designed. So, if you're going to pipe a response, you have to write headers, then pipe and watch for errors. You can't change that order.

I want to avoid writing headers until I know that the query won't produce an error before any rows

The only way to do that is to preload the entire response into memory (so that you know no errors have occurred while getting all the data) and then you can write the headers and then write the response (if no errors) and write different headers and a different response if there's an error.

If you're piping, then somewhat by definition, the headers have already been written and the data has probably started to be sent before an error occurs. This is a drawback of piping like you are using it. It has some advantages (like potentially lower memory usage and very simple coding), but there really isn't a great way to deal with errors when doing it that way. Without trying to get really, really fancy with multiple mime parts and allowing you to attach a final mime part to communicate more info about an error and creating your own piping solution that doesn't just hang up the socket on an error, but allows you to terminate the current mime part and add another one, if you're using piping, you're accepting the simplistic error communication that it provides.

jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • Thanks for the detailed answer. Based on the responses I think I did a poor job explaining what I'm trying to do. The case that I'm trying to handle is when there is an error before any rows are emitted. So if a column name is wrong for example, I want to be able to respond with a useful error message. I'm trying to avoid writing headers until I know the query will be returning some rows. The code example I gave shows an attempt at writing headers when the first row is emitted but that fails. – Surplus Yogurt Jul 11 '19 at 04:42
  • 1
    @SurplusYogurt - You're going to have to do the first part of the streaming manually then so you can see that you have some rows BEFORE you call `.pipe()` as `.pipe()` is too automatic to do that. I don't know enough about your sql stream to know how to do that. – jfriend00 Jul 11 '19 at 04:48