How to scale sharded Akka Cluster automatically?

Question

Akka Cluster documentation doesn't specify how we can automatically scale the cluster - adding or removing nodes. How is this supposed to be implemented in Akka ecosystem? For example, in two cases - when we're using VMs for nodes, or containers in Kubernetes composing the cluster.

If the load in a particular node is low, does Akka Cluster move the Actors from there to other nodes and shutdown the underutilized node automatically?

If Akka Cluster doesn't have elasticity capabilities, and relies on e.g. Kubernetes for that (as described here), then again if Kubernetes decides to remove an under-utilized Akka Cluster node, how would it be ensured that the Actors remaining in the under-utilized node will be 'gracefully' moved to other cluster nodes?

Answer 1

Akka Cluster doesn't handle moving actors from node to node.

Cluster sharding will move responsibility for hosting a shard as nodes leave and join the cluster. Basically, if a message is sent through cluster sharding to an actor in a shard which is unowned (e.g. because the node hosting that shard is no longer in the cluster), a node will be assigned that shard (by default, if I remember correctly, the strategy is to start the shard on the node responsible for the fewest shards, but other strategies, like starting the shard on the node where the initial message was sent (useful for things like ingest from a partitioned Kafka topic)).

When actors start on a new node, they will not by default retain their state from any node they previously ran on. Akka Persistence provides the ability to allow actors to recover their state; this is why it is very common to use Akka Persistence whenever Akka Cluster Sharding is in use.

Starting instances of a service is not a responsibility of Akka, it's the responsibility of Kubernetes, Mesos, or whatever control plane you care to use. As for scaling down, it's not exceptionally difficult to have your cluster nodes take themselves offline if they're not busy: though you're somewhat on your own for implementing that (and will probably have to interact with k8s etc. to prevent a graceful exit from being interpreted as a failure requiring a new instance to be spawned).