Nodejs - How to prevent loss when node crashes?

Question

I have an express app where users can dynamically creates subdomains which proxy sites.

Users can potentially create thousands of unique proxies, but if the app errors & crashes or restarts then this data is lost.

var app = require('express')();
var proxy = require('express-http-proxy');
var vhost = require('vhost');

app.get('/create', function(req, res){
    app.use(vhost('subdomain.mysite.com', proxy('http://example.com')));
    res.send('Created');
});

app.listen(8080);

I know I could store these in a database then loop through and recreate each one, but that doesn't seems like a solid solution for potentially thousands of uniquely created proxies.

I know these newly created routes are stored on the app variable (app._router). Is there a way to get routes from another source?(redis? mongo?)

Or is there a way to persist this routing information?

Or do any of the node management tools (PM2, forever, supervisor etc. ) prevent against or recover from this type of thing?

Or is there a better solution? Any advise is appreciated.

Answer 1

Clearly you need to persist this information in a way which survives your server process crash (which you seem to already know). The usual way of persisting information like this is to save it to a persistent store (usually a disk).

Architecturally, you have a couple ways of doing this:

1. Keep your current architecture, but add a "save to disk" step anytime the state of proxies is changed (one is added or removed or modified). So, anytime a change it made, you write your entire current state of routes to disk. To do this, you would add a new RAM-based data structure that holds your current state of all routes you've created. You could try to read this from Express, but frankly, I'd rather maintain my own data structure for this since it allows me to keep exactly what attributes I want. This will allow you, upon server startup, to read your saved configuration/state file and loop through the saved state to recreate the state you last had. If you save this state each time a change is made, the worst you can lose is the operation that was about to be saved. You will have to make sure you have appropriate concurrency protection in your save operation so two saves don't step on each other. That is not hard to do.

2. Switch to a database architecture, so each time you make a change, you write that one change to a database (a database using a persistent store). Then at anytime your server restarts, you can then read the state from the database and recreate what you had before the server restart.

The database is likely more scalable, but the write whole state to disk upon every change is likely simpler (no database required, just write your state to disk (probably JSON would be simplest to read and write). Either can work just fine up to a certain scale after which the database solution makes more sense (scale in terms of either changes per second or total number of proxies to keep track of).

I know I could store these in a database then loop through and recreate each one, but that doesn't seems like a solid solution for potentially thousands of uniquely created proxies.

I think you have this backwards. A database is probably more scalable for large numbers of proxies, though thousands is not a particularly large number. You can probably handle that size with either of the above techniques.

I know these newly created routes are stored on the app variable (app._router). Is there a way to get routes from another source?(redis? mongo?)

If this were my app, I would persist the data myself to a permanent store each time a new proxy was added, removed or modified and not use Express for keeping track of anything.

Or is there a way to persist this routing information?

Express doesn't have any features I'm aware of for automatically persisting routes. It is assumed in the express architecture that your startup code or subsequent code will create the route handlers it needs. You can persist the information yourself as you create these routes.

Or do any of the node management tools (PM2, forever, supervisor etc.) prevent against or recover from this type of thing?

Not that I know of. Those tools help manage the process itself, but not its internal state.

Or is there a better solution? Any advise is appreciated.

Persist the data yourself and then recreate the state from the persisted store upon server startup as described in the above two options.

---

Here's an example of the first option above that just saves the data each time a change is made and then upon server startup, reads the saved state:

const app = require('express')();
const proxy = require('express-http-proxy');
const vhost = require('vhost');
const Promise = require('bluebird');
const fs = Promise.promisify(require('fs'));

let proxyData = [];
readProxyData();

app.get('/create', function(req, res){
    app.use(vhost('subdomain.mysite.com', proxy('http://example.com')));

    // save proxy data
    proxyData.push({subDomain: 'subdomain.mysite.com', userDomain: 'http://example.com'})
    saveProxyData();

    res.send('Created');
});

app.listen(8080);

// call this anytime a new proxy has been added 
// (after the proxy info is added to the proxyData data structure)
function saveProxyData() {
     // use a promise to automatically sequence successive saves
     // makes a pending save wait for the current one to finish
     proxyData.promise = proxyData.promise.then(function() {
         return fs.writeFileAsync("proxyState.json", JSON.stringify(proxyData));
     }).catch(function(err) {
         // log save state error, but allow promise to continue so
         // subsequent saves will continue uninterrupted
         console.err(err);
         return;
     });
}

// only to be called upon server startup
function readProxyData() {
    try {
        proxyData = require("proxyState.json");
    } catch(err) {
        console.err("Error reading proxyState.json - continuing with no saved state: ", err);
    }
    // set initial promise state (used for chaining consecutive writes)
    proxyData.promise = Promise.resolve();

    // establish any previously existing proxies saved in the proxyData
    proxyData.forEach(function(item) {
        app.use(vhost(item.subDomain, proxy(item.userDomain)));
    });
}

Answer 2

I think you just need to caught unhandled error/exception. You may process uncaughtException that is your app won't exit if any error occured

process.on('uncaughtException', function (err) {
  console.log('Caught exception: ' + err);
});

Answer 3

Once a process ends, all data held in its working memory is lost. A common solution is to write that data to a more persistent system (database, file system, etc) as soon as it's received by your server.

I recently wrote an npm package cashola that tries to make this as easy as possible.

This is how your example might look using cashola.

var cashola = require('cashola');
var app = require('express')();
var proxy = require('express-http-proxy');
var vhost = require('vhost');

var state = cashola.rememberArraySync('state');

for (const item of state) {
    app.use(vhost(item.domain, proxy(item.proxy)));
}

app.get('/create', function(req, res){
    const vhostDomain = 'subdomain.mysite.com';
    const proxyDomain = 'http://example.com';
    state.push({ domain: vhostDomain, proxy: proxyDomain });
    app.use(vhost(vhostDomain, proxy(proxyDomain)));
    res.send('Created');
});

app.listen(8080);