Akka cluster detecting Quarantined state

Question

How can I detect Quarantined state from within the system being quarantined~?

I'm seeing this log below:

[warn] Remoting - Tried to associate with unreachable remote address [akka.tcp://Application@192.168.0.15:6000]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.

but I'm not sure how I can react to this from code.

I found this thread: Recovering from the Quarantined state, suggesting to listen for the QuarantinedEvent but that is not dispatched on the system being quarantined.

And I actually listened for all RemotingLifecycleEvents and found this:

AssociationError [akka.tcp://Application@192.168.0.100:2552] -> [akka.tcp://Application@192.168.0.15:6000]: Error [Invalid address: akka.tcp://Application@192.168.0.15:6000] [akka.remote.InvalidAssociation: Invalid address: akka.tcp://Application@192.168.0.15:6000 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

but that is just an AssociationError which will be dispatched for many other reasons as well, do I have to search for the actual text "The remote system has quarantined this system." within the error to be sure??

in flink's usage of akka, you'll find it matches the actual text to decide whether it is quarantined. — Bargitta, Aug 22 '18 at 02:39

score 7 · Answer 1 · answered Sep 09 '15 at 13:22

Yes, what you suggested works and can be done as follows

Subscribe an actor to akka.remote.AssociationErrorEvent

override def preStart(): Unit = {
  context.system.eventStream.subscribe(self, classOf[akka.remote.AssociationErrorEvent])
}

and then do following in the receive method

override def receive: Receive = {
  case e:AssociationErrorEvent =>
    log.info(s"AssociationErrorEvent: $e")
    if (e.cause.getCause.getMessage.contains("quarantined this system")) {
      log.warning(s"We got quarantined")
    }
}

Since 2.4.2 there is the `ThisActorSystemQuarantinedEvent` ([source](https://github.com/akka/akka/blob/v2.4.2/akka-remote/src/main/scala/akka/remote/RemotingLifecycleEvent.scala#L92)). — William Carter, Dec 04 '18 at 18:42

score 3 · Answer 2 · answered Dec 05 '18 at 08:52

Following William Carter comment:

Akka provides appropriate event that does not require check of the message content.

class MyActor extends Actor with ActorLogging {

  override def preStart(): Unit = {
    super.preStart()
    context.system.eventStream.subscribe(context.self, classOf[ThisActorSystemQuarantinedEvent])
  }

  override def receive: Receive = {
    case ev: ThisActorSystemQuarantinedEvent => log.warning(s"I was quarantined by ${ev.remoteAddress}")
  }
}

Akka cluster detecting Quarantined state

2 Answers2