How to avoid saturation in Akka HTTP (latency spikes)?

Question

I have a akka-http (Scala) API server that serves data to a NodeJS server. In moments after startup, everything works fine, everything is fast. Latency is low. But suddenly, latency increases fastly. The API no longer responds, and the website becomes unusable.

The strange thing is that the traffic and the requests count remain stable. Latency spikes seem decorrelated from them.

I guess this saturation is achieved with the blocking of all the threads in akka thread pool. Unfortunately, my Akka dispatcher is blocking, because I'm doing a lot of SQL queries (in MySQL) and because I'm not using a reactive library. I'm using Slick 2, which is, contrary to Slick 3, blocking-only. Here's my dispatcher :

http-blocking-dispatcher {
  type = Dispatcher
  executor = "thread-pool-executor"
  thread-pool-executor {
    fixed-pool-size = 46
  }
  throughput = 1
}

So, my question is, how to avoid this sort of bottling ? How to keep a latency proportional with the traffic ? Is there a way to evict the requests that cause the saturation in order to prevent them to compromise everything ?

Thank you !

Kinda tough to understand what's going on without looking at some code. I recommend you provide a mcve https://stackoverflow.com/help/mcve — Stefano Bonetti, Mar 25 '17 at 22:18

score 2 · Accepted Answer · answered Mar 27 '17 at 02:28

You should not use Akka's own thread pool for running long blocking tasks. Create your own thread pool, and run your slick queries using it, leaving free threads for akka. That's for your first 2 questions.

I don't know of any good answer for the last one. You could maybe look into specific slick settings to set a timeout on sql queries, but I don't know if such things exist. Otherwise try to analyse why your queries takes so much time, could you be missing an index or two?

How to avoid saturation in Akka HTTP (latency spikes)?

1 Answers1