0

I am using Spring Boot with Redis-Cache, with Lettuce default configuration, and receiving the following RejectedExecutionException after the server has been up for a few minutes:

org.springframework.data.redis.RedisSystemException: Unknown redis exception; nested exception is java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker
        at org.springframework.data.redis.FallbackExceptionTranslationStrategy.getFallback(FallbackExceptionTranslationStrategy.java:53)
        at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:43)
        at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
        at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.convertLettuceAccessException(LettuceStringCommands.java:718)
        at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:63)
        at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:210)
        at org.springframework.data.redis.cache.DefaultRedisCacheWriter.lambda$get$1(DefaultRedisCacheWriter.java:109)
        at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:242)
        at org.springframework.data.redis.cache.DefaultRedisCacheWriter.get(DefaultRedisCacheWriter.java:109)
        at org.springframework.data.redis.cache.RedisCache.lookup(RedisCache.java:82)
        at org.springframework.cache.support.AbstractValueAdaptingCache.get(AbstractValueAdaptingCache.java:58)
        at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:73)
        at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:525)
        at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:490)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:372)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:316)
        at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
        at my.controller.MyController.lambda$myFunc$0(MyController.java:60)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.concurrent.RejectedExecutionException: Thread limit exceeded replacing blocked worker
        at java.util.concurrent.ForkJoinPool.tryCompensate(ForkJoinPool.java:2011)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3310)
        at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
        at io.lettuce.core.protocol.AsyncCommand.await(AsyncCommand.java:81)
        at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:112)
        at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:62)
        at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
        at com.sun.proxy.$Proxy168.get(Unknown Source)
        at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:61)
        ... 21 common frames omitted

It seems that Lettuce uses the common ForkJoinPool, that the DeferredResult uses as well, and all the requests and connections are choking the pool (please correct me if I'm wrong). What is the recommended approach? Should I move Lettuce to use a different pool? If so how? Please let me if there is any other configuration or other information that I can provide.

apines
  • 1,254
  • 14
  • 36

1 Answers1

4

Lettuce uses Netty's EventLoop's as its threading infrastructure.

What here happens is that your task is executed on a ForkJoin pool. Lettuce uses CompletableFutures to return a handle for an async result processing and the synchronous API calls CompletableFuture.get(timeout, TimeUnit) to await command completion. Calling a blocking method on a ForkJoin pool involves the ManagedBlocker to potentially switch to a different thread that can proceed with work.

If too many threads await command completion, you eventually end up with RejectedExecutionException.

I suggest using a different execution pool for the lambda that you're invoking.

mp911de
  • 17,546
  • 2
  • 55
  • 95