In Node.js, why should I prefer key-value stores over application variable?

Question

I am developing a Socket.IO-backed real-time application in Node.JS that will be used by couple of hundred users simultaneously at any given moment, and I need to store some basic details about each connected client (and remove those details upon client disconnections).

I have read that using a key-value store (such as Redis) is the preferred choice for storing such data.

Why is storing data in a regular in-app variable (object, e.g. var connectedClientsData = {}) bad compared to storing data in a key-value store such as Redis?

Is it only to support scaling (e.g. multiple NodeJS-based application servers can connect to a single, central key-value store), or are there some more serious drawbacks?

What happens to the data when you restart your server? (It gets destroyed. This is why we use redis) — Raynos, Sep 20 '11 at 13:59

score 4 · Accepted Answer · answered Sep 20 '11 at 13:23

There are a couple of issues at play:

1) Yes, scaling. Not just to multiple servers, but also to multiple CPUs via something like "cluster" (which you can find on npm).

2) V8 has memory limitations currently based on its browser-based heritage. They are tweakable to some extent, and being removed more thoroughly in the next version of V8, but it's still something to consider.

Also you're much more likely to restart your node process (to update your application) than you are to restart redis, thus losing all the sessions.

score 2 · Answer 2 · answered Sep 20 '11 at 16:49

Data persistence - your data are persisted in more reliable and durable storage which is built to deal with this stuff. For example redis has snapshots of the dataset on disk and supports advanced data structures which offers you more options when manipulating with your data.
Data centralization - your data are stored and available from central place where not only your main application but also other apps can access it and manipulate with these data. For example if you need to share state/data between two systems it's much simpler to do so with KV store dedicated for this purpose.
Unloading your main application - when you use KV store you are unloading your main application which can improve it's performance and significantly lower it's memory consumption. Then scaling your main application can be much simpler.
KV store scalability - many KV stores has built-in scaling capabilities which would be otherwise more complex to implement in your main application (for example sharing state between multiple instances of your application). This functionality is also extensively tested and well documented so you don't have to care much about these things.

score 0 · Answer 3 · answered Sep 20 '11 at 13:20

It is said that key-value stores are better for storing persistent data, comparing to relational databases, not comparing to non-persisted in-memory objects.

But I suspect that a key-value database has advantages when compared to object variables, even when persistence is not a need. They can be shared among apps and can spare the memory consumption. Even Redis, which stores all values in memory, can save you memory from your application server because it can be migrated to another server when consuming a lot of gigabytes.

Also, an in-memory, shared object tends to become complex to maintain over time. If you decide to use it, I recommend you to abstract its nature. Have a object where you just do something like this in your code:

var storage = new Storage();
storage.registerClientDisconnection("client_id_1");

Then, your Storage.registerClientDisconnection() can be either

Storage.proptotype.registerClientDisconnection = function(clientId) {
    this.info[clientId].connected = false;
};

or

Storage.proptotype.registerClientDisconnection = function(clientId) {
    var client = redis.createClient();
    client.set("clientId:"+clientId+":connected", false);
};

Summarizing: I would recommend to use some key-value store but I bet you could use some object storage, too. However, if you decide to use an in-memory object, just abstract this implementation detail so you can easily migrate for a key-value store implementation in the future. Abstracting your decision seems more important than your decision about the implementation.

In Node.js, why should I prefer key-value stores over application variable?

3 Answers3