Using RedisSentinelConfiguration doesn't fail over to the next available sentinel/master

Question

I'm trying to setup HA for my redis instances using redis-sentinel. I assume my java application has to be using RedisSentinelConfiguration for creating a connection factory. Hence I have setup the connection factory like below

@Bean(name = "redisConnectionFactory")
    public JedisConnectionFactory redisConnectionFactory() {
        String redisHost = redisSystemProperties.getProperty("redis.host");
        int redisPort = redisSystemProperties.getIntegerProperty("redis.port", DEFAULT_REDIS_PORT);
        logger.info(format("Creating redis connection factory.  host: %s, port: %d", redisHost, redisPort));
        JedisConnectionFactory connectionFactory = new JedisConnectionFactory(redisSentinelConfig(), jedisPoolConfig());
        boolean usePool = Boolean.valueOf(redisSystemProperties.getProperty("pool.enabled", "true"));
        connectionFactory.setUsePool(usePool);
        connectionFactory.afterPropertiesSet();
        return connectionFactory;
    }

    @Bean(name ="redisSentinelConfig")
    public RedisSentinelConfiguration redisSentinelConfig(){
        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration();
        for (int i = 0; i < MAX_REDIS_MASTER_NODES; i++) {
            String hostPropertyName = format("redis.%d.host", i);
            String host = redisSystemProperties().getProperty(hostPropertyName);
            String portPropertyName = format("redis.%d.port", i);
            String portStr = redisSystemProperties().getProperty(portPropertyName);
            if (StrUtils.anyBlank(host, portStr)) {
                break;
            }
            int port = toInt(portStr);
            Assert.isTrue(port > 0, format("Invalid value '%s' for property '%s'", portStr, portPropertyName));
            RedisNode redisNode = new RedisNode(host, port);
            logger.info(format("Adding connection to redis sentinel [%d]: host: '%s', port: %d", i, host, port));
            sentinelConfig.addSentinel(redisNode);
            sentinelConfig.setMaster("mymaster");
        }
        return  sentinelConfig;
    }

The IP and port numbers that are used to create RedisSentinelConfiguration are of the sentinel instances I have started. I have started a sentinel for each master. I have 3 masters running so does 3 sentinels. The application seems to be working great as long as the master node which is mapped to sentinel 'mymaster' is running. Once I SHUTDOWN that redis node the application fails to get the connection. I see connection refused error. While debugging I can see that it is trying to create a connection using the single Sentinel configuration that it used in the first attempt.

Here is my sentinel configuration looks like.

port 26379

sentinel monitor mymaster 127.0.0.1 7000 2
sentinel down-after-milliseconds mymaster 1
sentinel failover-timeout mymaster 1
sentinel config-epoch mymaster 0
# Generated by CONFIG REWRITE
dir "/Users/skandula/redis-3.0.2"
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster 127.0.0.1 26380 e4bc16a7435eec64512acc03404beb9799dea73e
sentinel known-sentinel mymaster 127.0.0.1 26381 493d598ce30dd4429bdc53c94cf297b8a0436c67
sentinel current-epoch 0

Can someone let me know what could I be doing wrong? I'm using spring-data-redis 1.5.0.RELEASE and Jedis-2.7.0

Thank you

score 1 · Accepted Answer · edited Jun 24 '15 at 13:59

1

There's a gap between failover and the actual implementation to pick up the newly elected master node. Although Sentinel itself might be configured to failover immediately, the java driver implementation will likely be a bit behind. You'll need to take care of those situations.

Running the sentinel example you can easily explore the behavior by randomly shutting down Redis and/or Sentinel instances, while watching the command line printing out errors and how long recovery actually took.

You could also use spring-retry to reschedule attempts of writing to redis. There's a ticket (DATAREDIS-370) open for integrating RetryTemplate - so you might want to vote for it.

edited Jun 24 '15 at 13:59

mp911de

17,546
2
55
95

answered Jun 23 '15 at 06:03

Christoph Strobl

6,491
25
33

I think I have same exact configuration in my application too. I do see the sentinel is lost the connection and retrying but it looks like it doesn't try with the available sentinels. This is what I see in the log "SEVERE: Lost connection to Sentinel at 127.0.0.1:26379. Sleeping 5000ms and retrying. Jun 23, 2015 10:25:32 AM redis.clients.jedis.JedisSentinelPool$MasterListener run SEVERE: Lost connection to Sentinel at 127.0.0.1:26379. Sleeping 5000ms and retrying." Do you see anything wrong with my sentinel config? – Srini Kandula Jun 23 '15 at 14:30
I figured this. It is actually my configuration that is wrong. I didn't need to do the clustering. Redis replication and enough sentinel processes to moniter the server is what I needed. And yes spring-retry really handy. I'll try to make a blog post and link it here. Thank you Chris. Spring ROCKS. – Srini Kandula Jun 23 '15 at 16:45
BTW when I upgraded to spring-data-redis 1.0.5 my gradle build removed jedis dependency. I would get a compilation error for not finding JedisPoolConfig. I had to explicitly add that dependency to my build. – Srini Kandula Jun 23 '15 at 17:09

Using RedisSentinelConfiguration doesn't fail over to the next available sentinel/master

1 Answers1