0

I'm building a RESTful API using Akka and Spray. I'm using the Typesafe Akka Cluster Sample project...

(http://www.typesafe.com/activator/template/akka-sample-cluster-scala)

as a reference. I have a cluster which uses a pool of routees and cluster aware routing. The interface to the API is through a spray frontend which creates an actor (ClusterClient) that acts as a frontend to the cluster and monitors cluster activity (new nodes, downed nodes etc...). When a request comes in to a spray route I delegate the work to a Cluster backend worker via the cluster aware router.

My hope/goal here is that I can increase performance in the system by adding more back-end worker nodes and routees as needed. I may also add more frontend (spray) nodes that will be accessed via a Load Balancer. I'm trying to build something performant and scalable.

This all works fine. The worker actors at the moment just return a string for testing.

Before I go any further I wanted to get an idea of current cluster performance by stress testing the application. So I am using apache bench to do some quick tests.

All the testing is being done on my local development machine (QuadCore Macbook Pro with 16gb RAM). The application is running on spray-can. The frontend (spray) and backend nodes are being kicked off on seperate JVM's on the same machine by using different ports.

The strange thing is performance seems to degrade when I add more backend nodes to the cluster. Here are my tests and results. I ran each test about 10 times to get better average...

One Backend Node

ab -l -n 1000 -c 100 http://localhost:3050/api

Requests per second [#/sec] (mean): 4708.25, 5086.11, 4341.52, 5850.76, 4403.33, 5046.78, 4151.84, 5686.24, 3399.38, 4466.26, 3755.98, 5619.18, 5187.96
Average: 4745 requests per second

Two Backend Nodes

ab -l -n 1000 -c 100 http://localhost:3050/api

Requests per second [#/sec] (mean): 3381.75, 4108.51, 2969.41, 3732.29, 3804.48, 3644.12, 2516.05, 2968.17, 3308.01, 3378.69, 4049.71, 3738.21, 4068.84
Average: 3512 requests per second

Three Backend Nodes

ab -l -n 1000 -c 100 http://localhost:3050/api

Requests per second [#/sec] (mean): 2756.49, 2516.77, 2535.34, 2464.37, 2887.51, 2421.98, 2794.03, 2633.36, 2735.74, 3021.81, 2612.39, 3052.53, 3714.88
Average: 2780 requests per second

Is this expected? i.e. Am I wrong in thinking that I should be getting more performance by adding more backend/worker nodes? Based on the figures above, the more nodes I add to the cluster the worse the performance will become. Is this possibly an issue to do with running the tests on the same machine?

Thanks

fatlog
  • 1,182
  • 2
  • 14
  • 28
  • 1
    If you run more nodes on the same machine, there will be more cluster coordination work necessary between nodes, so the same machine with limited processing power will have less time to actually answer the requests so performance is expected to get worse. However, it seems to degrade more than I would have guessed. It would be interesting to see how it looks like in a real cluster. Also it would be interesting to see how much time is actually spent doing actual work vs. coordinating things in the cluster. You will need to profile to find out. – jrudolph May 07 '15 at 12:20
  • Hi, thanks for the feedback. I'll see what I can do to deploy this in a more real environment. With regards to profiling. Have you any suggestions of good tools to do this? Would/Should I use tools like Kamon and Gatling or are you thinking standard JVM profiling? Also, would you generally expect to see performance increases when clustering an application? – fatlog May 07 '15 at 13:40
  • Hi, So I deployed my app into a more realistic environment on AWS. I had two instances running (only t2.small instances). Each had the backend app running which provide the cluster workers. One of the instances also had a spray frontend running which handles incoming requests. Using apache bench showed the same degradation - a drop of about 30% in number of requests per second when running two backend nodes compared to one. Am I using Akka's clustering capabilities incorrectly? Thanks – fatlog May 13 '15 at 14:31
  • This could be anything, unfortunately. If you want advice you need to include much more detail and a reproducible example with instructions how to run it and what to expect and why. – jrudolph May 17 '15 at 11:29
  • Hi, thanks for the feedback jrudolph. I appreciate there is little to go on here. For the moment I've dropped using Akka's clustering features. However, I did read on a few posts that this could be an issue with message serialisation across the cluster. Could that be a cause? – fatlog May 20 '15 at 15:05
  • I'm no expert on cluster things so I'd suggest you post your question on the akka-user mailing list at https://groups.google.com/forum/#!forum/akka-user where it is more likely you get proper answer. – jrudolph May 20 '15 at 15:13

0 Answers0