Actor supervised by BackoffSupervisor loses stashed messages after restart

Question

I have an actor with stash usage. Sometimes, when it crashes, it loses all stashed messages. I found that it depends on what supervision logic I use.

I wrote a simple example.

An actor with the stash:

case object WrongMessage
case object TestMessage
case object InitialMessage

class TestActor extends Actor with Stash {
  override def receive: Receive = uninitializedReceive

  def uninitializedReceive: Receive = {
    case TestMessage =>
      println(s"stash test message")
      stash()

    case WrongMessage =>
      println(s"wrong message")
      throw new Throwable("wrong message")

    case InitialMessage =>
      println(s"initial message")
      context.become(initializedReceive)
      unstashAll()
  }

  def initializedReceive: Receive = {
    case TestMessage =>
      println(s"test message")
  }
}

In the following code, TestActor never receives stashed TestMessage:

object Test1 extends App {
  implicit val system: ActorSystem = ActorSystem()
  val actorRef = system.actorOf(BackoffSupervisor.props(Backoff.onFailure(
      Props[TestActor], "TestActor", 1 seconds, 1 seconds, 0
  ).withSupervisorStrategy(OneForOneStrategy()({
    case _ => SupervisorStrategy.Restart
  }))))
  actorRef ! TestMessage
  Thread.sleep(5000L)
  actorRef ! WrongMessage
  Thread.sleep(5000L)
  actorRef ! InitialMessage
}

But this code works well:

class SupervisionActor extends Actor {
  val testActorRef: ActorRef = context.actorOf(Props[TestActor])

  override def supervisorStrategy: SupervisorStrategy = OneForOneStrategy()({
    case _ => SupervisorStrategy.Restart
  })

  override def receive: Receive = {
    case message => testActorRef forward message
  }
}

object Test2 extends App {
  implicit val system: ActorSystem = ActorSystem()
  val actorRef = system.actorOf(Props(classOf[SupervisionActor]))
  actorRef ! TestMessage
  Thread.sleep(5000L)
  actorRef ! WrongMessage
  Thread.sleep(5000L)
  actorRef ! InitialMessage
}

I looked into sources and found that actor supervision uses LocalActorRef.restart method which backed by system dispatcher logic, but BackoffSupervisor simply creates a new actor after termination of the old one. Is there any way to work around it?

score 1 · Accepted Answer · answered Dec 28 '18 at 17:34

I'm not sure one can make restart under BackoffSupervisor properly send stashed messages without some custom re-implementation effort.

As you've already pointed out that BackoffSupervisor does its own restart that bypasses the standard actor lifecycle. In fact, it's explicitly noted in the BackoffOnRestartSupervisor source code:

Whatever the final Directive is, we will translate all Restarts to our own Restarts, which involves stopping the child.

In case you haven't read about this reported issue, it has a relevant discussion re: problem with Backoff.onFailure.

Backoff.onStop would also give the wanted BackoffSupervisor feature, but unfortunately it has its own use cases and won't be triggered by an exception.

Thank you for links. It seems like stashing just doesn't work with BackoffSupervisor. — Aleksey Isachenkov, Jan 05 '19 at 01:45

Actor supervised by BackoffSupervisor loses stashed messages after restart

1 Answers1