Performance issue sending larger messages via akka remote actors

Question

The responses to concurrent requests to remote actors were taking long time to respond, aka 1 request takes 300 ms, but 100 concurrent requests took almost 30 seconds to complete! So it almost looks like the requests are being executed sequentially! The request size is small, but response size was about 120 kB in JVM before serialization. But the response had deep nested case class.

The response times are similar when running on two different JVMs on same machine as well. But responses are fast when in same JVM (i.e. local actors). It is a single client making concurrent requests to one remote actor.

I see this log in akka debug logs. What does this indicate?

DEBUG test-app akka.remote.EndpointWriter - Drained buffer with maxWriteCount: 50, fullBackoffCount: 546, smallBackoffCount: 2, noBackoffCount: 1 , adaptiveBackoff: 2000

JVS · Accepted Answer · 2017-05-20T03:53:03.353

The logs show that write to send-buffer failed. This could indicate that

send-buffer is too small
receive-buffer on the remote actor's side is too small
network issues

The send buffer size and receive buffer size directly limits the number of concurrent requests and responses! Increase the send buffer and receive buffer sizes, on both client and server, to support the required concurrency in both client and server.

If the buffer size is not adequate, netty will wait for the buffer to be cleared before attempting to rewrite to the buffer. And by default there will be a backoff time too, and this can be configured as well.

The settings are under remote.netty.tcp:

akka {
 remote {
  netty.tcp {
   # Sets the send buffer size of the Sockets,
   # set to 0b for platform default
   send-buffer-size = 1024000b

   # Sets the receive buffer size of the Sockets,
   # set to 0b for platform default
   receive-buffer-size = 2048000b
  }
  # Controls the backoff interval after a refused write is reattempted.
  # (Transports may refuse writes if their internal buffer is full)
  backoff-interval = 1 ms
 }
}

For full configuration see Akka reference config.

Performance issue sending larger messages via akka remote actors

1 Answers1