7

I'm trying to understand how Slick-Hikari works, I've read a lot of documentation but I've a use case whose behavior I don't understand.

I'm using Slick 3 with Hikari, with the default configuration. I already have a production app with ~1000 users connected concurrently. My app works with websockets and when I deploy a new release all clients are reconnected. (I know it's not the best way to handle a deploy but I don't have clustering at the moment.) When all these users reconnect, they all starts doing queries to get their user state (dog-pile effect). When it happens Slick starts to throw a lot of errors like:

java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2@4dbbd9d1 rejected from java.util.concurrent.ThreadPoolExecutor@a3b8495[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 23740]

What I think it's happening is that the slick queue for pending queries is full because it can't handle all the clients requesting information from the database. But if I see the metrics that Dropwizard provides me I see the following:

Example of Observed Dropwizard Metrics

Near 16:45 we se a deploy. Until old instance is terminated we can see that the number of connections goes from 20 to 40. I think that's normal, given how the deploy process is done.

But, if the query queue of Slick becomes full because of the dog-pile effect, why is it not using more than 3-5 connections if it has 20 connections available? The database is performing really well, so I think the bottleneck is in Slick.

Do you have any advice for improving this deploy process? I have only 1000 users now, but I'll have a lot more in few weeks.

Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
SergiGP
  • 669
  • 7
  • 17
  • Maybe attaching a thread dump showing where all of those 20 supposedly "active" threads in the ThreadPoolExecutor are would help? Are they blocked on the HikariCP? Are they blocked on something else? Also, what version (exactly) of Slick and HikariCP? – brettw Sep 17 '16 at 10:01
  • Does this help http://stackoverflow.com/questions/29897003/slick-3-0-rc3-fails-with-java-util-concurrent-rejectedexecutionexception? – brettw Sep 17 '16 at 10:19
  • What version of Slick and HikariCP? I know Slick made some changes to try to increase concurrency in the past few months... – brettw Oct 26 '16 at 12:34

1 Answers1

0

Based on the "rejected" exception, I think many slick actions were submitted to slick concurrently, which exceeded the default size(1000) of the queue embedded in slick.

So I think you should:

  1. increase queue size(queueSize) to hold more unprocessed actions.
  2. increase number of thread(numThreads) in slick to process more actions concurrently. You can get more tips here
Community
  • 1
  • 1
qilinma
  • 1