Handling Faults in Akka actors

Question

I've a very simple example where I've an Actor (SimpleActor) that perform a periodic task by sending a message to itself. The message is scheduled in the constructor for the actor. In the normal case (i.e., without faults) everything works fine.

But what if the Actor has to deal with faults. I've another Actor (SimpleActorWithFault). This actor could have faults. In this case, I'm generating one myself by throwing an exception. When a fault happens (i.e., SimpleActorWithFault throws an exception) it is automatically restarted. However, this restart messes up the scheduler inside the Actor which no longer functions as excepted. And if the faults happens rapidly enough it generates more unexpected behavior.

My question is what's the preferred way to dealing with faults in such cases? I know I can use Try blocks to handle exceptions. But what if I'm extending another actor where I cannot put a Try in the super class or some case when I'm an excepted fault happens in the actor.

import akka.actor.{Props, ActorLogging}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import akka.actor.Actor

case object MessageA

case object MessageToSelf


class SimpleActor extends Actor with ActorLogging {

  //schedule a message to self every second
  context.system.scheduler.schedule(0 seconds, 1 seconds, self, MessageToSelf)

  //keeps track of some internal state
  var count: Int = 0

  def receive: Receive = {
    case MessageA => {
      log.info("[SimpleActor] Got MessageA at %d".format(count))
    }
    case MessageToSelf => {
      //update state and tell the world about its current state 
      count = count + 1
      log.info("[SimpleActor] Got scheduled message at %d".format(count))

    }
  }

}


class SimpleActorWithFault extends Actor with ActorLogging {

  //schedule a message to self every second
  context.system.scheduler.schedule(0 seconds, 1 seconds, self, MessageToSelf)

  var count: Int = 0

  def receive: Receive = {
    case MessageA => {
      log.info("[SimpleActorWithFault] Got MessageA at %d".format(count))
    }
    case MessageToSelf => {
      count = count + 1
      log.info("[SimpleActorWithFault] Got scheduled message at %d".format(count))

      //at some point generate a fault
      if (count > 5) {
        log.info("[SimpleActorWithFault] Going to throw an exception now %d".format(count))
        throw new Exception("Excepttttttiooooooon")
      }
    }
  }

}


object MainApp extends App {
  implicit val akkaSystem = akka.actor.ActorSystem()
  //Run the Actor without any faults or exceptions 
  akkaSystem.actorOf(Props(classOf[SimpleActor]))

  //comment the above line and uncomment the following to run the actor with faults  
  //akkaSystem.actorOf(Props(classOf[SimpleActorWithFault]))

}

Answer 1

The correct way is to push down the risky behavior into it's own actor. This pattern is called the Error Kernel pattern (see Akka Concurrency, Section 8.5):

This pattern describes a very common-sense approach to supervision that differentiates actors from one another based on any volatile state that they may hold.

In a nutshell, it means that actors whose state is precious should not be allowed to fail or restart. Any actor that holds precious data is protected such that any risky operations are relegated to a slave actor who, if restarted, only causes good things to happen.

The error kernel pattern implies pushing levels of risk further down the tree.

See also another tutorial here.

So in your case it would be something like this:

SimpleActor 
 |- ActorWithFault

Here SimpleActor acts as a supervisor for ActorWithFault. The default supervising strategy of any actor is to restart a child on Exception and escalate on anything else: http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html

Escalating means that the actor itself may get restarted. Since you really don't want to restart SimpleActor you could make it always restart the ActorWithFault and never escalate by overriding the supervisor strategy:

class SimpleActor {
  override def preStart(){
    // our faulty actor --- we will supervise it from now on
    context.actorOf(Props[ActorWithFault], "FaultyActor") 
  ...

  override val supervisorStrategy = OneForOneStrategy () {
    case _: ActorKilledException => Escalate
    case _: ActorInitializationException => Escalate
    case _ => Restart // keep restarting faulty actor
  }

}

Answer 2

To avoid messing up the scheduler:

class SimpleActor extends Actor with ActorLogging {

  private var cancellable: Option[Cancellable] = None

  override def preStart() = {
    //schedule a message to self every second
    cancellable = Option(context.system.scheduler.schedule(0 seconds, 1 seconds, self, MessageToSelf))
  }

  override def postStop() = {
    cancellable.foreach(_.cancel())
    cancellable = None
  }
...

To correctly handle exceptions (akka.actor.Status.Failure is for correct answer to an ask in case of Ask pattern usage by sender):

...
def receive: Receive = {
    case MessageA => {
      try {
        log.info("[SimpleActor] Got MessageA at %d".format(count))
      } catch {
        case e: Exception =>
          sender ! akka.actor.Status.Failure(e)
          log.error(e.getMessage, e)
      }
    }
...