Improving Akka Remote Throughput

Question

We're thinking about using Akka for client server communications, and trying to benchmark data transfer. Currently we're trying to send a million messages where each message is a case class with 8 string fields.

At this point we're struggling to get acceptable performance. We see about 600KB/s transfer rate and idle CPUs on client and server, so something is going wrong. Maybe it's our netty configuration.

This is our akka config

Server {
  akka {
    extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
    loggers = ["akka.event.slf4j.Slf4jLogger"]
    loglevel = "DEBUG"
    logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
    log-dead-letters = 10
    log-dead-letters-during-shutdown = on
    actor {
      provider = "akka.cluster.ClusterActorRefProvider"
    }
    remote {
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "instance.myserver.com"
        port = 2553
        maximum-frame-size = 1000 MiB
        send-buffer-size = 2000 MiB
        receive-buffer-size = 2000 MiB
      }
    }
    cluster {
      seed-nodes = ["akka.tcp://server@instance.myserver.com:2553"]

      roles = [master]
    }
    contrib.cluster.receptionist {
      name = receptionist
      role = "master"
      number-of-contacts = 3
      response-tunnel-receive-timeout = 300s
    }
  }

}

Client {
  akka {
    loggers = ["akka.event.slf4j.Slf4jLogger"]
    loglevel = "DEBUG"
    logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
    actor {
      provider = "akka.remote.RemoteActorRefProvider"
    }
    remote {
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "127.0.0.1"
        port = 0
        maximum-frame-size = 1000 MiB
        send-buffer-size = 2000 MiB
        receive-buffer-size = 2000 MiB
      }
    }
    cluster-client {
      initial-contacts = ["akka.tcp://server@instance.myserver.com:2553/user/receptionist"]
      establishing-get-contacts-interval = "10s"
      refresh-contacts-interval = "10s"
    }
  }

}

UPDATE:

In the end, notwithstanding the discussion about serialisation (see below) we just converted our payloads to use byte arrays, that way serialisation would not affect the tests. We found that on core i7 with 24gb ram using jeroMQ (ie zeroMQ re-implemented in java - so still not the fastest) we saw consistently about 200k msgs/sec or about 20 MB/sec whilst on raw akka (ie without zeroMQ plugin) we saw about 10k msgs/sec or just under 1MB/sec. Trying on akka + zeroMQ made the performance worse.

@william-rubens can you share the final results of your test? — Tolsi, Oct 09 '15 at 06:56

score 4 · Accepted Answer · answered Aug 18 '15 at 08:02

To make it easy to get started with remoting akka uses Java serialization for message serialization, this is not what you'd typically use in production because it's not very fast and does not handle message versioning well.

What you should do is use kryo or protobuf for serialization and you should be able to get much better numbers.

You can read about how here, there are also a few links to available serializers at the bottom of the page: http://doc.akka.io/docs/akka/current/scala/serialization.html

Thank you for the clarification :) – William Rubens Aug 18 '15 at 09:50 — William Rubens, Aug 18 '15 at 09:50

score 0 · Answer 2 · answered Aug 17 '15 at 10:06

Ok here's what we discovered:

Using our client server model we were able to get about < 3000 msg/sec without doing anything, which was not really acceptable to us, but we were really confused with what was happening because we weren't able to max out the cpu.

Therefore we went back to the akka source code and found a benchmarking sample there:

sample.remote.benchmark.Receiver

sample.remote.benchmark.Sender

These are two classes that use akka remote to ping pong themselves a bunch of messages from two jvms on the same machine. Using this benchmark we were able to get around 10~15k msg/sec on corei7 with 24g, using about 50% cpu. We found that adjusting the dispatchers and threads allocated to netty made a difference, but only marginally. Using asks instead of tells, made it a bit slower, but not by much. Using a balancing pool of many actors to send and receive made it considerably slower.

For comparison we did a similar test using jeromq and we managed to get about 100k msg/sec using 100% CPU (same payload, same machine).

We also hacked the Sender and Receivers to use the Akka zeromq extension to pipe messages directly from one jvm to another bypassing akka remote all together. In this test we found that we could get fast send and receive to start with, approx 100k msg/sec, but that performance quickly degraded. On further inspection the zeromq threads and the akka actor switching do not play nice together. We probably could have tuned the performance somewhat by being smarter with how zeromq and akka interacted with each other, but at that point we decided it would be better to go with raw zeromq.

Conclusion: Don't use akka if you care about fast serialisation of lots of data over the wire.

William, Akka does not implement a serializer - if you don't configure a proper serializer (which is highly discouraged) you end up using Java Serialization, which is widely known to be slow. So your benchmark is bottleneck-ed by Java Serialization not by Akka itself. Please refer to http://doc.akka.io/docs/akka/current/scala/serialization.html to read how to configure serializers and pick a fast one, like Kryo or ProtoBuf or something of that kind. Here's how to use Kryo https://github.com/romix/akka-kryo-serialization — Konrad 'ktoso' Malawski, Aug 18 '15 at 08:21
sorry it's a bit confusing. There's an UPDATE in the original question, those are the results when we just sent strings across the wire so serialisation was no longer an issue — William Rubens, Oct 28 '15 at 10:13

Improving Akka Remote Throughput

2 Answers2