How to add seed nodes to a running akka-cluster

Question

I am trying to create a fault tolerant akka-cluster which will be deployed in AWS. Its a standard cluster with 3 seed-nodes SN1, SN2 and SN3 and multiple Akka-systems connected to one seed-node. The SN's have ASG's which will spin a new instance in case one goes down.

I know that if SN1 fails the system will failover to the other SN (seed-nodes). But how can I register SN1 which now has a new IP address seamlessly without stopping the cluster.

I tried the following with no luck

• Cluster.joinseednodes only works on startup
• ELB behind each SN did not work. Akka couldn't connect to the ELB

Does anyone have ideas as to how one can add seed-nodes to a running Akka-cluster?

Thanks

Answer 1

Please check this:

http://chrisloy.net/2014/05/11/akka-cluster-ec2-autoscaling.html

Is in scala, but is using autoscalling too and explains about how to find your new IP (S1) and the rest of ips (S2...) and joining them.

Hope will help.

Answer 2

We ended up registering the ip address:port to zookeeper via a zookeeper-seed plugin. The plugin implements a latch if I am not mistaken.

Basically Each aka system gets the list of systems from zookeeper. It then removes itself from the list and uses the new list as its seed nodes.

If I get the time I will try to write up something on the lessons learned implementing a 24 system akka cluster.

Thanks

Answer 3

The answer to that question is NOT, you can't add seed nodes to a running Akka-cluster.

From Akka scala-doc:

An actor system can only join a cluster once. Additional attempts will be ignored. When it has successfully joined it must be restarted to be able to join another cluster or to join the same cluster again.

BUT you can start node and that connect to one of already created seed nodes and will be added to the cluster dynamically.

An example in Akka documentation is here: https://doc.akka.io/docs/akka/current/cluster-usage.html#a-simple-cluster-example