node.js and redis cluster: handling disconnections

Question

TL;DR Using ioredis with a redis cluster, what is the correct way to get an error if redis becomes unreachable, and then try to reconnect?

More details:

I'm using ioredis to connect to a redis cluster from node.js and trying to create a reasonably robust solution for disconnections. What I want to achieve is on the one hand, to get an error when a request to redis failed, while on the other hand, attempting to reconnect in the background and on the next request. Ioredis doesn't seem to document this very well, and the only way I'm able to achieve this is by combining enableOfflineQueue=true, adding a clusterRetryStrategy that returns false, and when catching the connection error, creating a new connection by re-instantiating ioredis using new Redis.Cluster(...).

Note that no other configuration I have tried achieves this behavior: Either it retries forever without ever throwing an error (if clusterRetryStrategy is not defined) or it throws an error immediately and will never again try to reconnect (if enableOfflineQueue is false), regardless of what I define for reconnectOnError.

My main question is, is this a good strategy? Am I leaving behind uncleaned up resources in the previous Redis.Cluster instance (i.e., will this create a memory leak)? Is there a better way to achieve this kind of behavior (for example, explicitly telling ioredis to reconnect, without re-instantiating)?
Another question, if this technique is actually the recommended strategy, is: It seems to take quite a long time until I get an error (5-10 seconds running against a local cluster) - is there a way to make the time until I get a connection-error shorter? Is there a way to configure this time (to, say, ~2 seconds)?

For clarity, below is the code I'm using for testing this. The really relevant part of the code is just

if (redis.status === 'end') {
    redis = connectToCluster();
}

which is what the question is really about.

Following is the code, which I run in combination with a local redis cluster (on windows WSL), which I start and stop while the test is running to observe the behavior.

const Promise = require('bluebird');
const Redis = require('ioredis');

function connectToCluster() {
    return new Redis.Cluster([
        { host: 'localhost', port: 30001 },
        { host: 'localhost', port: 30002 },
        { host: 'localhost', port: 30003 },
        { host: 'localhost', port: 30004 },
        { host: 'localhost', port: 30005 },
        { host: 'localhost', port: 30006 },
    ], {
        enableOfflineQueue: true,
        clusterRetryStrategy: times => false
    });
}

let redis = connectToCluster();

describe('Redis client: ', function () {
    it('should error when redis fails and return correct connection state when connected or not', async function () {
        this.timeout(300000);
        try {
            await wait(500);
            try {
                for (let i = 0; i < 100; i++) {
                    await redis.set(i, i);
                }
            } catch (e) {
                console.log(`======================================= failed to insert test data =======================================`);
            }

            let succeeded = 0, failed = 0;
            for (let i = 0; i < 300000; i++) {
                let start = Date.now();
                try {
                    if (redis.status === 'end') {
                        console.log(`======================================= RECONNECTING TO CLUSTER (redis.status = "${redis.status}") =======================================`);
                        redis = connectToCluster();
                    }

                    await redis.get(i % 100);
                    succeeded++;
                } catch (e) {
                    console.log(`======================================= request ${i} failed in ${Date.now() - start}ms =======================================`);
                    failed++;
                }

                if (i % 1000 === 0) {
                    console.log(`=============================== retrieved ${i} times, succeeded: ${succeeded}, failed: ${failed} ===============================`);
                    global.gc();
                }
            }
        } catch (e) {
            console.log(`========== caught error: ${e.stack}`);
        }
    });
});

async function wait(ms = 10) {
    await Promise.resolve().delay(ms);
}

node.js and redis cluster: handling disconnections

0 Answers0