We have a Redis cluster(running in Sentinel mode) with 3 nodes and we are using Redisson (version 3.10.7) to interact with Redis. We are facing an issue wherein the client application is not able to connect to the cluster. Continuously older connections are being dropped and new connections are being created. The same thing is happening every second. Below is the snapshot for the logger.
12:01:00.832 [redisson-netty-2-6] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@483495011 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0x767f3c87, L:/192.168.7.24:60173 - R:10.79.9.163/10.79.9.163:6379], command=null]
12:01:00.832 [redisson-netty-2-6] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@1510307492 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0xa8c01128, L:/192.168.7.24:60183 - R:10.79.9.163/10.79.9.163:6379], command=null]
12:01:00.832 [redisson-netty-2-6] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@425141862 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0x19afdfb8, L:/192.168.7.24:60199 - R:10.79.9.164/10.79.9.164:6379], command=null]
12:01:00.832 [redisson-netty-2-6] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@457747719 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0xf2431dfb, L:/192.168.7.24:60211 - R:10.79.9.164/10.79.9.164:6379], command=null]
12:01:00.832 [redisson-netty-2-6] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@1595469651 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0x3226faec, L:/192.168.7.24:60204 - R:10.79.9.164/10.79.9.164:6379], command=null]
12:01:00.922 [pool-2-thread-1] DEBUG org.redisson.client.handler.ConnectionWatchdog - reconnecting RedisConnection@541871189 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0x5396b42d, L:/192.168.7.24:59969 ! R:10.79.9.164/10.79.9.164:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@3ecd3a79(failure: java.util.concurrent.CancellationException)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]] to 10.79.9.164/10.79.9.164:6379
12:01:00.929 [pool-2-thread-1] DEBUG org.redisson.client.handler.ConnectionWatchdog - reconnecting RedisConnection@495642036 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0xfa2f4f52, L:/192.168.7.24:59973 ! R:10.79.9.164/10.79.9.164:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@28c788eb(failure: java.util.concurrent.CancellationException)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]] to 10.79.9.164/10.79.9.164:6379
12:01:00.931 [pool-2-thread-1] DEBUG org.redisson.client.handler.ConnectionWatchdog - reconnecting RedisConnection@169281839 [redisClient=[addr=redis://10.79.9.164:6379], channel=[id: 0xf365bc68, L:/192.168.7.24:59976 ! R:10.79.9.164/10.79.9.164:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@6bd0d406(failure: java.util.concurrent.CancellationException)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]] to 10.79.9.164/10.79.9.164:6379
12:01:00.931 [pool-2-thread-1] DEBUG org.redisson.client.handler.ConnectionWatchdog - reconnecting RedisConnection@65640722 [redisClient=[addr=redis://10.79.9.165:6379], channel=[id: 0x03c68dcf, L:/192.168.7.24:59226 ! R:10.79.9.165/10.79.9.165:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@520c88b3(failure: java.util.concurrent.CancellationException)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]] to 10.79.9.165/10.79.9.165:6379
12:01:00.932 [pool-2-thread-1] DEBUG org.redisson.client.handler.ConnectionWatchdog - reconnecting RedisConnection@1365249560 [redisClient=[addr=redis://10.79.9.165:6379], channel=[id: 0x0722fcc6, L:/192.168.7.24:59262 ! R:10.79.9.165/10.79.9.165:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@50c8b29f(failure: java.util.concurrent.CancellationException)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]] to 10.79.9.165/10.79.9.165:6379
12:01:01.080 [redisson-netty-2-8] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@1230181180 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0x148352d3, L:/192.168.7.24:60224 - R:10.79.9.163/10.79.9.163:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@19cb007f(incomplete)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]]
12:01:01.082 [redisson-netty-2-5] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@1221343292 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0xc4ff65cd, L:/192.168.7.24:60229 - R:10.79.9.163/10.79.9.163:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@1891a694(incomplete)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]]
12:01:01.082 [redisson-netty-2-7] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@2036916006 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0xff04b8ab, L:/192.168.7.24:60223 - R:10.79.9.163/10.79.9.163:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@1618e7eb(incomplete)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]]
12:01:01.082 [redisson-netty-2-9] DEBUG org.redisson.connection.ClientConnectionsEntry - new connection created: RedisConnection@1460667823 [redisClient=[addr=redis://10.79.9.163:6379], channel=[id: 0xbdc0d69e, L:/192.168.7.24:60222 - R:10.79.9.163/10.79.9.163:6379], command=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@139695d9(incomplete)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]]
We tried below points to investigate the issue:
- Restart the client application.
- Restart Redis cluster nodes.
Post these restarts as well, the same issue persisted.
We were able to resolve the issue by changing the timeout property from 0 to 10 seconds in server.conf of Redis. On reverting the timeout back to 0 also is not causing the issue to reoccur again.
Anyone else faced similar kind of issue?. If yes, do we know the reason behind it?
