6

I am using Akka.NET to implement an actor system in which some actors are created on demand and are deleted after a configurable idle period (I use Akka's "ReceiveTimeout" mechanism for this). Each of these actors is identified by a key, and there should not exist two actors with the same key.

These actors are currently created and deleted by a common supervisor. The supervisor can be asked to return a reference to the actor matching a given key, either by returning an existing one or creating a new one, if an actor with this key doesn't exist yet. When an actor receives the "ReceiveTimeout" message, it notifies the supervisor who in turn kills it with a "PoisonPill".

I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception. Worse, when sending an "Ask" message, the sender remains blocked, waiting indefinitely (or until a timeout) for a response that he will never receive.

I first thought about Akka's "Deatchwatch" mechanism to monitor an actor's lifecycle. But, if I'm not mistaken, the "Terminated" message sent by the dying actor will be received by the monitoring actor asynchronously just like any other message, so the problem may still occur in between the target actor's death and the reception of its "Terminated" message.

To solve this problem, I made it so that anyone asking the supervisor for a reference to such an actor has to send a "close session" message to the supervisor to release the actor when he doesn't need it anymore (this is done transparently by a disposable "ActorSession" object). As long as there are any open sessions on an actor, the supervisor will not delete it.

I suppose that this situation is quite common and am therefore wondering if there isn't a simpler pattern to follow to address this kind of problem. Any suggestion would be appreciated.

Odsh
  • 697
  • 5
  • 20

2 Answers2

3

I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception.

This is by design. You will never receive an exception upon attempting to send a message - it will simply be routed to Deadletters and logged. There's a lot of reasons for this that I won't get into here, but the bottom line is that this is intended behavior.

DeathWatch is the right tool for this job, but as you point out - you might receive a Terminated message after you already sent a message to that actor.

A simpler pattern than tracking open / closed sessions is to simply use acknowledgement / reply messages from the recipient using Ask + Wait + a hard timeout. The downside of course is that if your recipient actor has a lot of long-running operations then you might block for a long period of time inside the sender.

The other option you can go with is to redesign your recipient actor to act as a state machine and have a soft-terminated or terminating state that it uses to drain connections / references with potential senders. That way the original actor can still reply and accept messages, but let callers know that it's no longer available to do work.

Aaronontheweb
  • 8,224
  • 6
  • 32
  • 61
  • I thought about the solution with Ask + timeout, but I don't like the idea of my actors not doing anything because they are waiting for a timeout. I realize this will happen sooner or later because of the "at-most-sone" delivery reliability in Akka, but at least in that case it would be a message transport issue betsween the actors, not a design constraint. I like the "terminating" state idea, I will give it some more thought. However, the actor still doesn't know at what point it will not receive any message anymore and thus when he can delete itself. – Odsh Dec 23 '14 at 09:37
  • @Odsh "However, the actor still doesn't know at what point it will not receive any message anymore and thus when he can delete itself." - if the receiving actor doesn't know when it will stop receiving messages and the sending actor doesn't know when it must stop sending messages, then this is your design problem. One of these two actors must be able to make that determination or report back to some other actor who can. – Aaronontheweb Dec 24 '14 at 21:31
  • Aaron, the sending and receiving actors know when they will stop exchanging messages. However, the receiving actor doesn't know when a new actor will want to initiate a communication with it. – Odsh Dec 29 '14 at 08:25
  • 1
    Great answer, but I would add that you can also check for an Actor with Actorselection: "You can also acquire an ActorRef for an ActorSelection with the resolveOne method of the ActorSelection. It returns a Future of the matching ActorRef if such an actor exists. It is completed with failure akka.actor.ActorNotFound if no such actor exists or the identification didn't complete within the supplied timeout." – Chanan Braunstein Mar 19 '15 at 11:10
0

I solved this problem with entity actors created through Akka's Cluster Sharding mechanism:

If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of the entity actors for example by defining receive timeout (context.setReceiveTimeout). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such messages the entity actor can send ShardRegion.Passivate to its parent Shard. The specified wrapped message in Passivate will be sent back to the entity, which is then supposed to stop itself. Incoming messages will be buffered by the Shard between reception of Passivate and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity.

Odsh
  • 697
  • 5
  • 20
  • Did you need to implement all aspect of Cluster Sharding mechanism (going all through the doc), or just partially to solve this problem? – Minh Kha Aug 25 '21 at 04:40
  • That was a long time ago Minh Kha, I don't remember the specifics. If my memory serves me well, that was my only reason for using Cluster Sharding, so I kept to a minimum. – Odsh Aug 29 '21 at 07:50