4

I am writing an an Akka Http service (v2.4.2) service for talking to a cassandra instance. Running the service on my laptop, it appears not to scale with the number of simultaneous connections.

~/workspace/wrk $ ./wrk -t8 -c32 -d30s -s random-object.lua http://localhost:8080/buckets/newbucket
Running 30s test @ http://localhost:8080/buckets/newbucket
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    14.23ms   29.40ms 304.93ms   91.78%
    Req/Sec   703.35    226.18     1.31k    75.40%
  165252 requests in 30.03s, 27.74MB read
  Requests/sec:   5503.64
  Transfer/sec:      0.92MB
~/workspace/wrk $ ./wrk -t8 -c64 -d30s -s random-object.lua http://localhost:8080/buckets/newbucket
Running 30s test @ http://localhost:8080/buckets/newbucket
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    20.66ms   37.87ms 676.67ms   90.45%
    Req/Sec     0.86k   288.76     2.09k    72.99%
  199623 requests in 30.09s, 33.51MB read
  Requests/sec:   6633.58
  Transfer/sec:      1.11MB
~/workspace/wrk $ ./wrk -t8 -c128 -d30s -s random-object.lua http://localhost:8080/buckets/newbucket
Running 30s test @ http://localhost:8080/buckets/newbucket
  8 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.52ms   35.14ms 506.00ms   88.56%
    Req/Sec     0.87k   275.83     2.36k    72.13%
  207498 requests in 30.08s, 34.83MB read
  Socket errors: connect 0, read 96, write 2, timeout 0
  Requests/sec:   6897.64
  Transfer/sec:      1.16MB

Actually just noticed the socket errors once I break 64 connections. All Akka configs are default. If I look at the stats in visual VM, akka creates a bunch of new dispatchers to handle the load, but 90% of cpu time is spent here in akka code (it appears): 

"metadata-service-akka.actor.default-dispatcher-62" #101 prio=5 os_prio=31 tid=0x00007fc261a7f800 nid=0xa503 waiting on condition [0x0000700003f50000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006c0098818> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
    at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
    - None
Dylan Wilder
  • 166
  • 1
  • 9
  • 2
    The thread you show does not run Akka code (at that moment), it is just idle. Your question lacks all detail that would be needed to help you. In particular “talking to cassandra” needs a lot more qualification. Another thing is that Akka does not start dispatchers, only you do that; Akka cannot magically know how to “handle the load”. – Roland Kuhn Apr 16 '16 at 09:11
  • This is akka-http and yes it does start dispatchers. These are indeed akka threads running in a fork join pool. No, none of the above code says "akka" (aside from the name of the pool), but akka is managing this pool and is solely responsible for allocating work to it. – Dylan Wilder Apr 16 '16 at 14:58
  • Yes, I know all that, but still you didn’t provide any information that would allow me to help – Roland Kuhn Apr 16 '16 at 18:35
  • What information would help? By talking to cassandra, I mean I'm using the datastax java driver with async calls only. I don't see latency increasing on the cassandra calls. I'm running no actors of my own, they are all created by the http framework (high level routing dsl). If you have more specific questions, I could answer them. – Dylan Wilder Apr 20 '16 at 20:59

0 Answers0