I am using a redis cluster of node type m5.4x large with 1 node , in-order to cache some results. The writes to this redis node is very frequent. And I could see that intermittently the writes to cluster fails.
Below is the stack trace we see in logs for the failure.
org.redisson.client.WriteRedisConnectionException: Unable to send command! Node source: NodeSource[slot=null,addr=null,redisClient=null,redirect=null,entry=org.redisson.connection.MasterSlaveEntry@6608962a], connection: [id: 0xbad70cba, L:0.0.0.0/0.0.0.0:47904], command: (EVAL),params: [local insertable = false; local value = redis.call('hget',KEYS[1], ARGV[5]); local t, val;if value ..., 8, SEARCH_CACHE, redisson__timeout__set:{SEARCH_CACHE}, redisson__idle__set:{SEARCH_CACHE}, redisson_map_cache_created:{SEARCH_CACHE}, redisson_map_cache_updated:{SEARCH_CACHE}, redisson__map_cache__last_access__set:{SEARCH_CACHE}, redisson_map_cache_removed:{SEARCH_CACHE}, {SEARCH_CACHE}:redisson_options, ...] at org.redisson.command.CommandAsyncService.checkWriteFuture(CommandAsyncService.java:675) at org.redisson.command.CommandAsyncService.access$100(CommandAsyncService.java:84) at org.redisson.command.CommandAsyncService$9$1.operationComplete(CommandAsyncService.java:638) at org.redisson.command.CommandAsyncService$9$1.operationComplete(CommandAsyncService.java:635) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1371) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081) at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.channels.ClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
I am using redisson client version 3.6.5. Can someone please help me to identify what is the issue?
Below is the configuration I have setup for redis cluster connection
- idleConnectionTimeout: 1000
- pingTimeout: 10000
- connectTimeout: 10000
- timeout: 30000
- retryAttempts: 3
- retryInterval: 1500
- reconnectionTimeout: 30000
- failedAttempts: 3
- subscriptionsPerConnection: 5
- slaveSubscriptionConnectionMinimumIdleSize: 1
- slaveSubscriptionConnectionPoolSize: 50
- slaveConnectionPoolSize: 250
- masterConnectionMinimumIdleSize: 5
- masterConnectionPoolSize: 250
