Stop New Request to the Express server but can process older req and send response

Question

I am working on a graceful shutdown. My app has APIs that take 10 -20 sec to respond

server.close(
    () => {
      log('HTTP server closed')
    }
  );

The above code works when there is a time gap between the API call, but it never stops my server as my server gets new requests before responding to the older requests.

score 2 · Answer 1 · answered Nov 11 '21 at 06:02

The node .close() method does almost exactly what you are asking it to do. Specifically:

server.close([callback])
Added in: v0.1.90
callback Called when the server is closed.
Returns: <net.Server>
Stops the server from accepting new connections and keeps existing connections.

The issue you may have is that someone is sending new requests over an existing connection that is using keepalive. No new connections are actually being created, but the effect is the same.

The only way I have found to solve this is to actively track all open connections as well as each request on said connection and then when shutting down:

Forcibly close connections with no active requests connection.destroy()
Reject requests that are received on existing connections request.res.end()

Actual logic will look something like this, there's missing logic here but this should get you close enough to solve it.

const connections = [];
let shuttingDown = false;

server.on('connection', (conn) => {
  let connectionId = 'some unique id here';
  
  conn.connectionId = connectionId;
  conn.requests = []; // so we can track open requests
  
  conn.on('close' => {
    delete connections[connectionId];
  });
  
  connections[connectionId] = conn;  
});

server.on('request', (req) => {
  // I don't actually know if the req.connection.connectionId will exist here
  // due to possible race conditions, or immutability of the connection object
  // if that is the case you may need to find another way to determine a unique
  // identifier based on existing connection fields
  let conn = connections[req.connection.connectionId];
  conn.requests.push(req);
  
  function requestComplete() {
    // if connection still exists
    // logic here for deleting request from connection.requests array
    // if shutting down, and connection.requests.length = 0; then connection.end()
  }
  
  req.res.on('finish', requestComplete);
  req.res.on('close', requestComplete);
  req.res.on('end', requestComplete);
  
  // If the server is already shutting down before this request is received
  // We do this after adding listeners for the request in case this is the only
  // request for the connection, so that our existing logic will auto-close the
  // socket instead of needing to duplicate it here as a special case
  if (shuttingDown) {
    req.res.statusCode = 503;
    return req.res.end();
  }
});

function shutdown() {
  shuttingDown = true;
  server.close(() => {
    console.log('closed');
  });
}

Recommended upgrades:

logging open connections & their requests during shutdown if it takes longer than reasonable
Forcibly closing requests & connections after some pre-determined time (may want to log which ones get forced so you know if certain requests get stuck)
Check for any longpoll requests, these are indefinite and so must be forcibly killed (text/event-stream headers or a path you know only serves that content)

For the above method of keeping a list of connections will it be efficient for a very large no of req per second? — ankalal, Nov 11 '21 at 06:25
I have run something like this in a production environment at 150 requests per second per worker without issues. Haven't explicitly tested it beyond that limit. — Rob Riddle, Nov 12 '21 at 06:39

jfriend00 · Answer 2 · 2021-11-11T06:32:10.910

You can implement middleware that immediately rejects incoming connections as soon as you start your shutdown process.

// set this flag to true when you want to start 
//   immediately rejecting new connections
let pendingShutdown = false;

// first middleware
app.use((req, res, next) => {
   if (pendingShutdown) {
       // immediately reject the connection
       res.sendStatus(503);
   } else {
       next();
   }
});

With no long running connections as soon as the ones that were in process finish, the server should find a natural exit point when you do this:

pendingShutdown = true;
server.close();

There are also a few modules on NPM that provide various algorithms for shut-down too.

Then, to prevent any long lived stuck connections from preventing your server from shutting down, you can either set a timeout on the existing connections or just set a global timeout and do a process.exit() after your timeout (to force shutdown).

Stop New Request to the Express server but can process older req and send response

2 Answers2