4

I'm running a Tomcat application which uses Jedis to access a Redis database. Form time to time the whole application blocks. By monitoring Tomcat using JavaMelody I found out that the problem seems be related to the JedisPool when a object requests a Jedis instance.

catalina-exec-74
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
redis.clients.util.Pool.getResource(Pool.java:20)
....

This is the JedisPoolConfig I'm using 

JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxActive(20);
poolConfig.setTestOnBorrow(true);
poolConfig.setTestOnReturn(true);
poolConfig.setMaxIdle(5);
poolConfig.setMinIdle(5);
poolConfig.setTestWhileIdle(true);
poolConfig.setNumTestsPerEvictionRun(10);
poolConfig.setTimeBetweenEvictionRunsMillis(10000);
jedisPool = new JedisPool(poolConfig, "localhost");

So obviously some threads try to get a Jedis instance but the pool is empty and cannot return an instance so the default pool behavior is wait.

I've already double checked my whole code and I'm pretty sure I return every Jedis instance to the pool that I used before. So I'm not sure why I'm running out of instance.

Is there a ways to check how many instances are left in the pool? I'm trying to find a sensible value for the maxActive parameter to prevent the application from blocking.

Are there any other ways to create memory holes other than not returning the Jedis instances to the pool?

Flo
  • 27,355
  • 15
  • 87
  • 125
  • 1
    Did you manage to solve this issue? I'm having exactly the same problem and as you mention, I'm pretty sure all connections are returned (I'm using a DAO pattern, so there's just one class using Jedis). Thanks in advance. – pablochacin Jan 14 '13 at 16:01

3 Answers3

5

Returning the resource to the Pool is important, so remember to do it. Otherwise when closing your app it'll wait for the resource to return.

https://groups.google.com/forum/?fromgroups=#!topic/jedis_redis/UeOhezUd1tQ

After each Jedis method call, return the resource pool. Your app has probably used all the threads and waits for some to be dropped. This may cause behavior you're explaining and the app is probably blocked.

Jedis jedis = JedisFactory.jedisPool.getResource();
try{
    jedis.set("key","val");
}finally{
    JedisFactory.jedisPool.returnResource(jedis);
}
vladaman
  • 3,741
  • 2
  • 29
  • 26
3

Partial answer to hopefully be of some help to people in similar scenarios, though I'm unsure if my problem was the same as yours (if you've since figured it out, please let us know!).

I've already double checked my whole code and I'm pretty sure I return every Jedis instance to the pool that I used before. So I'm not sure why I'm running out of instance.

I thought I had to - I always put my code in try / finally blocks, but it turns out I did have a memory leak: 

Jedis jedis = DBHelper.pool.getResource();
    try {
        // Next line causes the leak
        Jedis jedis = DBHelper.pool.getResource();
        ...
    } finally {
        DBHelper.pool.returnResource(jedis);
    }

No idea how that second call snuck in, but it did and caused both a leak and the web app to block.

Is there a ways to check how many instances are left in the pool? I'm trying to find a sensible value for the maxActive parameter to prevent the application from blocking.

In my case, I both found the bug and optimized based on the number of clients seen by the redis server. I set the loglevel (in redis.conf to verbose (default is notice), which will report about every 5-10 seconds the number of clients connected. Once I found my memory leak, I repeatedly sent requests to the page calling that method, and watched the redis clients reported by redis logs climb, but never drop.. I would think that would be a good start for optimizing?

Anyways, hope this is helpful to someone!

chris
  • 2,404
  • 3
  • 27
  • 33
0

When you use jedis pool, every time you get the resource using getResource(), you have to call releaseResource(). And if number of threads are more than resources, you will have thread contention. I found it much simpler to have Jedis connection per thread using Java ThreadLocal. So, for each thread check whether jedis connection already exists. If yes, use it, otherwise create a connection for the running thread. This ensures there wouldn't be any lock contention or error conditions to look after.

Vineet Gupta
  • 82
  • 3