Akka don't elect existing worker node as seed node, once the seed node is killed

Question

I am a newbie with akka. Having a very simple configuration of one seed node and one non-seed node.

Please find below, sequence of operation :-

Seed node started :-

val pConfig = ConfigFactory.load("application.conf").getConfig("PrimaryNode")

val demoSystem = ActorSystem("DemoSystem", pConfig)

[INFO] [04/01/2017 23:42:48.438] [run-main-0] [akka.remote.Remoting] Starting remoting                                                                                                                                                          
[INFO] [04/01/2017 23:42:48.673] [run-main-0] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://DemoSystem@127.0.0.1:2551]                                                                                           
[INFO] [04/01/2017 23:42:48.688] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Starting up...                                                                                    
[INFO] [04/01/2017 23:42:48.759] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Registered cluster JMX MBean [akka:type=Cluster]                                                  
[INFO] [04/01/2017 23:42:48.759] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Started up successfully                                                                           
[INFO] [04/01/2017 23:42:48.776] [DemoSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Metrics will be retreived from MBeans, and may be incorrect on som
e platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar                     
[INFO] [04/01/2017 23:42:48.779] [DemoSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Metrics collection has started successfully                       
p: akka.actor.ActorSystem = akka://DemoSystem                                                                                                                                                                                                   

[INFO] [04/01/2017 23:42:48.795] [DemoSystem-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Node [akka.tcp://DemoSystem@127.0.0.1:2551] is JOINING, rol
es []                                                                                                                                                                                                                                           
[INFO] [04/01/2017 23:42:48.805] [DemoSystem-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://DemoSystem@127.0.0.1:2551] to [U
p]

Non-seed node started :-

val sConfig = ConfigFactory.load("application.conf").getConfig("SecondaryNode")
val secSystem = ActorSystem("DemoSystem", sConfig)

[INFO] [04/01/2017 23:44:15.530] [run-main-0] [akka.remote.Remoting] Starting remoting
[INFO] [04/01/2017 23:44:15.845] [run-main-0] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://DemoSystem@127.0.0.1:49169]
[INFO] [04/01/2017 23:44:15.847] [run-main-0] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://DemoSystem@127.0.0.1:49169]
[INFO] [04/01/2017 23:44:15.854] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Starting up...
[INFO] [04/01/2017 23:44:15.929] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [04/01/2017 23:44:15.929] [run-main-0] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Started up successfully
[INFO] [04/01/2017 23:44:15.945] [DemoSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
[INFO] [04/01/2017 23:44:15.951] [DemoSystem-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Metrics collection has started successfully
s: akka.actor.ActorSystem = akka://DemoSystem

[INFO] [04/01/2017 23:44:16.265] [DemoSystem-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Welcome from [akka.tcp://DemoSystem@127.0.0.1:2551]

And seed node moves non-seed node to up (seed node logs after starting non-seed node) :-

[INFO] [04/01/2017 23:44:16.190] [DemoSystem-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Node [akka.tcp://DemoSystem@127.0.0.1:49169] is JOINING, roles []

[INFO] [04/01/2017 23:44:16.782] [DemoSystem-akka.actor.default-dispatcher-25] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://DemoSystem@127.0.0.1:49169] to [Up]

Now seed node is killed (logs on non seed node says, that seed node is quaretined) :-

............
............

[WARN] [04/01/2017 23:46:14.301] [DemoSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp://DemoSystem@127.0.0.1:49169/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDemoSystem%40127.0.0.1%3A2551-2] Association with remote system [akka.tcp://DemoSystem@127.0.0.1:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://DemoSystem@127.0.0.1:2551]] Caused by: [Connection refused: no further information: /127.0.0.1:2551]
[INFO] [04/01/2017 23:46:14.959] [DemoSystem-akka.actor.default-dispatcher-21] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Leader is auto-downing unreachable node [akka.tcp://DemoSystem@127.0.0.1:2551]
[INFO] [04/01/2017 23:46:14.960] [DemoSystem-akka.actor.default-dispatcher-20] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Marking unreachable node [akka.tcp://DemoSystem@127.0.0.1:2551] as [Down]
[INFO] [04/01/2017 23:46:15.953] [DemoSystem-akka.actor.default-dispatcher-21] [akka.cluster.Cluster(akka://DemoSystem)] Cluster Node [akka.tcp://DemoSystem@127.0.0.1:49169] - Leader is removing unreachable node [akka.tcp://DemoSystem@127.0.0.1:2551]
[WARN] [04/01/2017 23:46:15.953] [DemoSystem-akka.remote.default-remote-dispatcher-5] [akka.remote.Remoting] Association to [akka.tcp://DemoSystem@127.0.0.1:2551] having UID [-273153546] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.

Now when i check the following things on non-seed node console :-

which node is the leader

Leader ship is successfully transferred to the non seed node

scala> val cluster = Cluster(secSystem)

scala> cluster.state
res1: akka.cluster.ClusterEvent.CurrentClusterState = CurrentClusterState(TreeSet(Member(address = akka.tcp://DemoSystem@127.0.0.1:49169, status = Up)),Set(),Set(akka.tcp://DemoSystem@127.0.0.1:49169),Some(akka.tcp://DemoSystem@127.0.0.1:49169),Map())

scala> cluster.state.leader
res2: Option[akka.actor.Address] = Some(akka.tcp://DemoSystem@127.0.0.1:49169)

which node is the seed node :-

Seed node still points to the dead node

scala> cluster.settings.SeedNodes
res0: scala.collection.immutable.IndexedSeq[akka.actor.Address] = Vector(akka.tcp://DemoSystem@127.0.0.1:2551)

I was thinking, that the non-seed node will be elected as a new seed node, but that was not the case.

Also, please find below configuration of two nodes :-

PrimaryNode {
  akka {
    actor {
      provider = "akka.cluster.ClusterActorRefProvider"
    }
    remote {
      log-remote-lifecycle-events = off
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "127.0.0.1"
        port = 2551
      }
    }
    cluster {
      seed-nodes = ["akka.tcp://DemoSystem@127.0.0.1:2551"]
      auto-down-unreachable-after = 10s
    }
  }
}

SecondaryNode {
  akka {
    actor {
      provider = "akka.cluster.ClusterActorRefProvider"
    }
    remote {
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "127.0.0.1"
        port = 49169
      }
    }
    cluster {
        seed-nodes = ["akka.tcp://DemoSystem@127.0.0.1:2551"]
        auto-down-unreachable-after = 10s
    }
  }
}

Please enlighten me. Why the non seed node was not elected as new seed node once the previous seed node was killed.

Where do the `cluster.state` and `cluster.state.leader` come from? Could you put more context around those to be sure the results are not outdated... — Frederic A., Apr 02 '17 at 08:32
@FredericA. : Updated question to include cluster and cluster.state.leader — mogli, Apr 03 '17 at 01:33

score 1 · Answer 1 · answered Apr 03 '17 at 06:01

Sorry that I have very little time now, I'll try my best in a few words.

There's no such thing as seed node election.

Once an akka cluster has booted, every cluster member can act as a seed node for a joining node i.e., be used as initial contact point by any joining node. And this happens without configuration as all cluster nodes listen for incoming connections.

Before a cluster has booted though, the (first) seed node is very important as it is required to be part of the initial nodes available when starting up the cluster. Without that node, the cluster won't be able to boot. But once booted (as said above), that seed node can disappear and everything will be still fine.

This is most likely why the value of cluster.settings.SeedNodes doesn't change or is not updated. It has no impact after initial cluster setup.

references:

Akka don't elect existing worker node as seed node, once the seed node is killed

1 Answers1