Error kernel and supervision in Akka: will messages from stale children be delivered to restarted actor?

Question

I am trying to follow best practices and apply error kernel pattern in Akka. According to this quote from here:

If one actor carries very important data (i.e. its state shall not be lost if avoidable), this actor should source out any possibly dangerous sub-tasks to children it supervises and handle failures of these children as appropriate. Depending on the nature of the requests, it may be best to create a new child for each request, which simplifies state management for collecting the replies. This is known as the “Error Kernel Pattern” from Erlang.

... it is a good idea to create children and delegate error prone work to them, concentrating important state in parent/supervisor actor.

In this scenario if actor with important state gets restarted for some reason, do I need to handle messages from it's stale children (that was created before restart)?

Lets illustrate this with example.

Lets asuume I have actors 3 actors: A (is parent/supervisor of B), B (is parent/supervisor of C, contains important state) and C.

A's supervision strategy is configured to restart it's children on exception.

C gets created in B's constructor.

Lets then assume that message bc is sent form B to C. C starts to process it (lets imagine that it runs long running computation there) and, once done will reply to B with cb.

Now, lets assume that before cb is sent to and processed by B A sends message ab to B. This message causes B to throw an exception and as a result of A's supervision strategy decision B will be restarted.

As a child of B C will be stopped during B's restart (new C' will be created in B's constructor).

Will restarted B receive cb from C that was sent before B got restarted?

If yes, will sender of cb (C) be considered a child of restarted B? And will actor ref's of C and C' be equal (assuming C and C's names are equal)?

score 2 · Accepted Answer · edited Jun 01 '15 at 16:44

Yes, your restarted B will receive its response from the first C. If C is performing long running work and its parent fails, the restart of B actually won't happen until C has finished its long running work. As part of restarting B, the original C is stopped (not restarted) and a new C is created (what you are calling C') that is a child of the restarted B. The first C will not be a child of the restarted B however.

When the first C finishes, its sender ref is still valid and this response can still be delivered to that refs mailbox even though it is about to be restarted. When B is restarted, it is able to preserve what was in its mailbox before the restart, thus it receives and processes the cb message immediately after starting back up.

I drafted a little code example to show this behavior:

import akka.actor._
import concurrent.duration._

object RestartExample extends App{
  case object Start
  case object AB
  case object BC
  case object CB

  val system = ActorSystem("test")
  val a = system.actorOf(Props[A])
  a ! Start

  class A extends Actor{
    val b = context.actorOf(Props[B], "myb")
    import context.dispatcher

    def receive = {
      case Start =>
        b ! Start
        context.system.scheduler.scheduleOnce(3 seconds, b, AB)
    }
  }

  class B extends Actor{
    println("new B actor instance created")
    val c = context.actorOf(Props[C], "myc")
    def receive = {
      case Start =>
        c ! BC
      case CB =>
        println("got a CB from C")
      case AB =>
        throw new RuntimeException("foo")
    }
  } 

  class C extends Actor{
    println("new C actor instance created")
    def receive = {
      case BC =>
        Thread.sleep(10000) // simulating long running behavior
        println("done with long processing...")
        sender ! CB
    }
  } 
}

Thanks for reply and code snippet. However, there are few corrections to be made before I can accept your answer. First of all, `C` never gets restarted. `C` gets `context.stop`'ed and another actor will be created in place of it (this is quite important difference since no restart hooks will be called on `C` and there will be new `C'` with different actor ref created by `B` after `B` restart). The second thing is answers to my question question about `C` and `C'` refs equality and whether `C` is a child of restarted `B`. The answer is no to both. — Eugene Loy, Sep 30 '14 at 08:26
I.e, when processing `cb` `sender == c` and `context.children.toSet.contains(sender)` are `false`. Ammend this info to your answer and I'll accept it. — Eugene Loy, Sep 30 '14 at 08:29

score 0 · Answer 2 · answered Sep 29 '14 at 12:50

Your question contains a contradiction:

"Now, lets assume that before cb is sent to and processed by B" "will restarted B receive cb from C that was sent before B got restarted"

So which one is it? Was cb sent before the restart or not?

If C restarts before sending the message, then it is lost. If it was sent before the restart, but not processed by B then it will be in B's mailbox and the new instance of B will receive it.

Basically, I would advice you to design a flow that does not fall apart as soon as one message is lost. When working with Akka, you should always assume that any message can be lost at any time.

Edit: forgot the second question: if cb was sent before the restart, and B actually receives it the sender will be an ActorRef pointing to the new instance of C. ActorRefs and Actors are 2 different entities.

Error kernel and supervision in Akka: will messages from stale children be delivered to restarted actor?

2 Answers2