How to cleanly/safely remove link from asynchronous communication chain

Question

I have a chain of Akka actors K -> L -> M, where K sends messages to L, L sends message to M, and each can reply to its predecessor using the sender of a received message.

My question is this: how can we safely unlink L from the chain, such that after the operation K sends messages directly to M, and M sees K as its sender?

If each actor was stable and long-lived, I could work out how to tell K and M to talk to each other. L could hang around long enough to forward messages already in her mailbox until getting a signal that each of K and M was no longer talking to L.

However, K and M might also be thinking about unlinking themselves, and that's where everything gets hairy.

Is there a well-known protocol for making this work safely? So far I haven't found the right search terms.

I know that one alternative would be to freeze the system and copy the entire thing, minus L, into a new chain. However, I would like to implement a more real-time, less interrupted solution.

I've been thinking about some kind of lock exchange, in which K and M promise to not unlink themselves until L has finished unlinking and forwarding all messages. But any solution with the word "lock" seems awkward in an asynchronous solution, even though the actors wouldn't be fully locked (just delaying their own unlink operations until a convenient time).

Answer 1

A rough idea for a solution would be to maintain state within each node for previous and next and then support messages for unkinking a node from the chain and also telling a node that it's next node has changed. When a next node changes, send a poison pill to the node that informed you of that (assuming it's the one being unlinked) to gracefully stop it. When unlinked and before stopped, act as a pure passthrough for any more messages that might come in. Putting all that together, the code would look something like this:

object ChainNode {
  case object Unlink
  case class Link(prev:Option[ActorRef], next:Option[ActorRef])
  case class ChangeNextNode(node:Option[ActorRef])
}

trait ChainNode extends Actor{
  import ChainNode._
  import context._

  override def postStop{
    println(s"${self.path} has been stopped")
  }

  def receive = chainReceive()

  def chainReceive(prevNode:Option[ActorRef] = None, nextNode:Option[ActorRef] = None):Receive = {
    case Unlink =>
      prevNode foreach{ node =>
        println(s"unlinking node ${self.path} from sender ${node.path}") 
        node ! ChangeNextNode(nextNode)
      } 
      become(unlinked(nextNode))

    case Link(prev, next) =>
      println(s"${self.path} is linking to $prev and $next")
      become(chainReceive(prev, next))

    case ChangeNextNode(newNext) =>
      println(s"${self.path} is changing next node to $newNext")
      become(chainReceive(prevNode, newNext))
      sender ! PoisonPill

    case other =>    
      println(s"${self.path} received message $other")
      val msg = processOther(other)
      nextNode foreach{ node =>
        println(s"${self.path} forwarding on to ${node.path}")
        node ! msg
      }
  }

  def unlinked(nextNode:Option[ActorRef]):Receive = {
    case any => 
      println(s"${self.path} has been unlinked, just forwarding w/o processing...")
      nextNode foreach (_ ! any)      
  }

  def processOther(msg:Any):Any
}

class NodeA extends ChainNode{
  def processOther(msg:Any) = "foo"
}

class NodeB extends ChainNode{
  def processOther(msg:Any) = "bar"
}

class NodeC  extends ChainNode{
  def processOther(msg:Any) = "baz"
}

Then, a simple test scenario where the linking is changed mid way through:

object ChainTest{
  import ChainNode._  

  def main(args: Array[String]) {
    val system = ActorSystem("chain")
    val a = system.actorOf(Props[NodeA])
    val b = system.actorOf(Props[NodeA])
    val c = system.actorOf(Props[NodeA])

    a ! Link(None, Some(b))
    b ! Link(Some(a), Some(c))
    c ! Link(Some(b), None)

    import system.dispatcher
    Future{
      for(i <- 1 until 10){
        a ! "hello"
        Thread.sleep(200)
      }
    }

    Future{
      Thread.sleep(300)
      b ! Unlink
    }
  }
}

It's not perfect, but it could serve as a good starting point for you. One downside is that messages like Unlink and ChangeNextNode still will be processed in the order they were received. If there are a bunch of messages in front of them in the mailboxes, those must be processed first before the change (unlinking for example) takes effect. This could lead to an unwanted delay in making the change. If that's an issue then you might want to look into a priority based mailbox where messages like Unlink and ChangeNextNode have a higher priority then the rest of the messages being handled.