How do I shut down my Express server gracefully when its process is killed?

Question

When running my Express application in production, I want to shut down the server gracefully when its process is killed (i.e. a SIGTERM or SIGINT is sent).

Here is a simplified version of my code:

const express = require('express');

const app = express();

app.get('/', (req, res) => res.json({ ping: true }));

const server = app.listen(3000, () => console.log('Running…'));

setInterval(() => server.getConnections(
    (err, connections) => console.log(`${connections} connections currently open`)
), 1000);

process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);

function shutDown() {
    console.log('Received kill signal, shutting down gracefully');
    server.close(() => {
        console.log('Closed out remaining connections');
        process.exit(0);
    });

    setTimeout(() => {
        console.error('Could not close connections in time, forcefully shutting down');
        process.exit(1);
    }, 10000);
}

When I run it and call the URL http://localhost:3000/ in a browser, the log statement in the setInterval function will keep printing “1 connection currently open” until I actually close the browser window. Even closing the tab will keep the connection open, apparently.

So when I kill my server by hitting Ctrl+C, it will run into the timeout and print “Could not close connections” after 10 seconds, all the while continuing to print “1 connection open”.

Only if I close the browser window before killing the process I get the “closed out remaining connections” message.

What am I missing here? What is the proper way to shut down an Express server gracefully?

Answer 1

I added a listener for connections opening on the server, storing references to those connections in an array. When the connections are closed, they are removed from the array.

When the server is killed, each of the connection is closed by calling its end methods. For some browsers (e.g. Chrome), this is not enough, so after a timeout, I call destroy on each connection.

const express = require('express');

const app = express();

app.get('/', (req, res) => res.json({ ping: true }));

const server = app.listen(3000, () => console.log('Running…'));

setInterval(() => server.getConnections(
    (err, connections) => console.log(`${connections} connections currently open`)
), 1000);

process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);

let connections = [];

server.on('connection', connection => {
    connections.push(connection);
    connection.on('close', () => connections = connections.filter(curr => curr !== connection));
});

function shutDown() {
    console.log('Received kill signal, shutting down gracefully');
    server.close(() => {
        console.log('Closed out remaining connections');
        process.exit(0);
    });

    setTimeout(() => {
        console.error('Could not close connections in time, forcefully shutting down');
        process.exit(1);
    }, 10000);

    connections.forEach(curr => curr.end());
    setTimeout(() => connections.forEach(curr => curr.destroy()), 5000);
}

Answer 2

The problem you are experiencing is that all modern browsers reuse single connection for multiple requests. This is called keep-alive connections.

The proper way to handle this is to monitor all new connections and requests and to track status of each connection (is it idle or active right now). Then you can forcefully close all idle connections and make sure to close active connections after current request is being processed.

I've implemented the @moebius/http-graceful-shutdown module specifically designed to gracefully shutdown Express applications and Node servers overall. Sadly nor Express, nor Node itself doesn't have this functionality built-in.

Here's how it can be used with any Express application:

const express = require('express');
const GracefulShutdownManager = require('@moebius/http-graceful-shutdown').GracefulShutdownManager;


const app = express();

const server = app.listen(8080);

const shutdownManager = new GracefulShutdownManager(server);

process.on('SIGTERM', () => {
  shutdownManager.terminate(() => {
    console.log('Server is gracefully terminated');
  });
});

Feel free to check-out the module, the GitHub page has more details.

Answer 3

There is open source project https://github.com/godaddy/terminus recommended by the creators of Express (https://expressjs.com/en/advanced/healthcheck-graceful-shutdown.html).

The basic example of terminus usage:

const http = require('http');
const express = require('express');
const terminus = require('@godaddy/terminus');

const app = express();

app.get('/', (req, res) => {
  res.send('ok');
});

const server = http.createServer(app);

function onSignal() {
  console.log('server is starting cleanup');
  // start cleanup of resource, like databases or file descriptors
}

async function onHealthCheck() {
  // checks if the system is healthy, like the db connection is live
  // resolves, if health, rejects if not
}

terminus(server, {
  signal: 'SIGINT',
   healthChecks: {
    '/healthcheck': onHealthCheck,
  },
  onSignal
});

server.listen(3000);

terminus has a lot of options in case you need server lifecycle callbacks (ie. to deregister instance from service registry, etc.):

const options = {
  // healtcheck options
  healthChecks: {
    '/healthcheck': healthCheck    // a promise returning function indicating service health
  },

  // cleanup options
  timeout: 1000,                   // [optional = 1000] number of milliseconds before forcefull exiting
  signal,                          // [optional = 'SIGTERM'] what signal to listen for relative to shutdown
  signals,                          // [optional = []] array of signals to listen for relative to shutdown
  beforeShutdown,                  // [optional] called before the HTTP server starts its shutdown
  onSignal,                        // [optional] cleanup function, returning a promise (used to be onSigterm)
  onShutdown,                      // [optional] called right before exiting

  // both
  logger                           // [optional] logger function to be called with errors
};

Answer 4

If you allow me, there is even a better solution that involves less work by using server-destroy package. Internally this package will terminate gracefully each connection and then allow the server to be "destroyed". In this case we ensure to definitively end the express application (and potentially start it again if we use a call function). This works for me using electron, and can potentially be ported to a standard server:

const express = require('express')
const { ipcMain } = require('electron')
const enableDestroy = require('server-destroy')
const port = process.env.PORT || 3000

export const wsServer = () => {
  try {
    let app = null
    let server = null

    const startServer = () => {
      if (app) {
        app = null
      }

      app = express()
      app.use(express.static('public'))
      app.use('/', (req, res) => {
        res.send('hello!')
      })

      server = app.listen(3000, () => {
        console.log('websocket server is ready.')
        console.log(`Running webserver on http://localhost:${port}`)
      })

      enableDestroy(server)
    }

    const stopServer = () => {
      if (server !== null) {
        server.destroy()
        app = null
        server = null
      }
    }
    const restartServer = () => {
      stopServer()
      startServer()
    }

    ipcMain.on('start-socket-service', (event) => {
      startServer()
      console.log('Start Server...')
      event.returnValue = 'Service Started'
    })

    ipcMain.on('stop-socket-service', (event) => {
      stopServer()
      console.log('Stop Server...')
      event.returnValue = 'Service Stopped'
    })

    ipcMain.on('restart-socket-service', () => {
      restartServer()
    })

  } catch (e) {
    console.log(e)
  }
}

Answer 5

Check out https://github.com/ladjs/graceful#express

const express = require('express');
const Graceful = require('@ladjs/graceful');

const app = express();
const server = app.listen();
const graceful = new Graceful({ servers: [server] });
graceful.listen();

Answer 6

Http terminator seems to be the 2022 solution that handles keep alive conections properly and force shutdown after some time https://www.npmjs.com/package/http-terminator

The main benefit of http-terminator is that:

• it does not monkey-patch Node.js API
• it immediately destroys all sockets without an attached HTTP request
• it allows graceful timeout to sockets with ongoing HTTP requests
• it properly handles HTTPS connections
• it informs connections using keep-alive that server is shutting down by setting a connection: close header
• it does not terminate the Node.js process

Answer 7

Try the NPM express-graceful-shutdown module, Graceful shutdown will allow any connections including to your DB to finish, not allow any fresh/new ones to be established. Since you are working with express that may be the module you are looking for, however a quick NPM search will reveal a whole list of modules suited to Http servers etc.