2

When we close the Jedis connection, it closes on client side but on server connection is still in ESTABLISHED mode. connections are never getting closed on server for the same client and it keeps growing. And we are getting redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException:

when we run netstat on client it shows only 4 connections alive (ESTABLISHED ) but when we do netstat on redis server we see 900 connection in ESTABLISHED state with the same client. CLIENT LIST on redis server also shows 900 connections idle for days.

CLIENT LIST on Sentinel shows all connection in idle state and cmd=subscribe

    127.0.0.1:6379> client list
    id=6204 addr=11.25.111.136:50290 fd=189 name= age=332651 idle=332651 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6112 addr=11.25.111.153:36690 fd=106 name= age=436390 idle=436390 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6347 addr=11.25.111.152:32798 fd=324 name= age=151429 idle=151429 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6150 addr=11.25.111.136:56118 fd=135 name= age=403572 idle=403572 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6033 addr=11.27.111.167:48448 fd=30 name= age=533471 idle=533471 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6321 addr=11.25.111.153:58718 fd=298 name= age=184226 idle=184226 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6378 addr=11.25.111.151:52860 fd=350 name= age=119895 idle=119895 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=6478 addr=11.23.111.138:47436 fd=437 name= age=1707 idle=1707 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe

Redis / Jedis Configuration

redis.with.sentinel = true
redis.pool.connection.timeout.millis=10000
redis.pool.max.total=100
redis.pool.max.idle=25
redis.pool.min.idle=10
redis.pool.max.wait.seconds=5000

Connection Pool for jedis sentinel

GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMaxTotal(maxTotal);
poolConfig.setMaxIdle(maxIdle);
poolConfig.setMinIdle(minIdle);
poolConfig.setMaxWaitMillis(maxWaitInSeconds*1000);
pool = new JedisSentinelPool(clusterName, sentinels,poolConfig,timeoutInMillis);

Client code to access resource from redis:

try (Jedis jedis = pool.getResource()) {
     jedis.get(key);
}catch (Exception e) {
}

Jedis version: 2.8.0 Redis version: Redis server v=3.0.3

Even after setting timeout value to 7200 in redis.conf, I am still seeing all those connections open in sentinel 6379. Timeout should have closed all idle connections only pub/sub are not closed but we are not using pub/sub.

Is there any other setting for JedisSentinel to close idle connections ?

enter image description hereI have observed that most of the open connections are for sentinel which is running on port 6379 on slave host in cluster

Please suggest why so many connections are in open state. We are not using any load balancers or proxy.

enter image description here

Sushil
  • 364
  • 6
  • 22
  • 1
    We are also seeing same issue. Did you find any root cause and solution to this? – AlwaysLearning Jul 13 '21 at 09:47
  • What we observed was that we have 2 regions setup and when ever there was cross region connection for redis those connections were not getting closed and immediately if we send any request that was failing due to connection limit. We did couple of changes like increasing timeout and retry after 10 sec. And retry actually solved our connection failure issue. This was not a correct solution but was a workaround as I could not find any solution from Jedis support – Sushil Jul 14 '21 at 17:24
  • We have observed this::: from client service, connection to redis-sentinel was either 1 or 0 but there were around 900+ connections estd from redis-sentinel to client machine. Also, when we stopped our client service and after 5 mins checked the connections from redis-sentinel machine to client, then also it was showing as 900+. Connection with redis(not sentinel) was matching to min idle count which is okay. Issue seems to be with redis-sentinel. – AlwaysLearning Jul 15 '21 at 18:33

0 Answers0