Losing event publishing in Persistent Actor on crash

Question

In this example from Akka persistance documentation

    val receiveRecover: Receive = {
    case evt: Evt                                 => updateState(evt)
    case SnapshotOffer(_, snapshot: ExampleState) => state = snapshot
    }

    val snapShotInterval = 1000
    val receiveCommand: Receive = {
    case Cmd(data) =>
      persist(Evt(s"${data}-${numEvents}")) { event =>
        updateState(event)
        context.system.eventStream.publish(event)
        if (lastSequenceNr % snapShotInterval == 0 && lastSequenceNr != 0)
          saveSnapshot(state)
      }
    case "print" => println(state)
    }

I understand that this lambda:

    event =>
    updateState(event)
    context.system.eventStream.publish(event)
    if (lastSequenceNr % snapShotInterval == 0 && lastSequenceNr != 0)
      saveSnapshot(state)

Is executed when the event has been successfully persisted. What if the actor crashes while this lambda is being executed BEFORE successful publishing of the event, ie before context.system.eventStream.publish(event)?

Do I understand correctly that in such case the event is never published which may lead to an inconsistent state of the system? If so, is there any way to detect that such thing happened?

[EDIT]

Also, if you use the event publishing in your system, then correct me if I am wrong:

1. If your application is deployed in one JVM and you use the default Akka's event publishing facilities, then JVM crash will mean that all events published but not yet processed will be lost since that facility does not have any recovery mechanisms.

2. If your application is deployed in a cluster then you'll run in the same situation as above only if the whole cluster goes down.

3. For any production setup you should configure something like Kafka for event publishing/consuming.

Answer 1

I understand that this lambda:

...

Is executed when the event has been successfully persisted. What if the actor crashes while this lambda is being executed BEFORE successful publishing of the event, ie before context.system.eventStream.publish(event)?

The lambda is run after the state is persisted. And the actor essentially suspends itself (putting all pending work in the stash) until that persistence is complete so that it remains consistent.

Do I understand correctly that in such case the event is never published which may lead to an inconsistent state of the system?

No, it will remain consistent for the above reason.

If your application is deployed in one JVM and you use the default Akka's event publishing facilities, then JVM crash will mean that all events published but not yet processed will be lost since that facility does not have any recovery mechanisms.

I guess it depends on what you mean by default event publishing. Regular actors, yes. If you lose the JVM you lose "regular" actors. Regular actors are in memory, essentially like normal Java/Scala objects. Persistent Actors, are, of course a different story.

You also say "published but not yet processed". Those, of course, are lost as well. Anything that isn't "processed" is essentially like a JDBC statement that hasn't been received by the database yet, or a message not transmitted to Kafka, etc. The design is essentially to save the event to the database immediately (almost like a transaction log) and then do the work after it is known to be safely persisted.

If your application is deployed in a cluster then you'll run in the same situation as above only if the whole cluster goes down.

A cluster essentially just gives a place for the persistent actor to recover. The cluster still relies on the persistent store for recovery.

(I'm keeping this answer focused on Akka Persistent Actors, the answers get more varied with things like Distributed Data.)

For any production setup you should configure something like Kafka for event publishing/consuming.

Not necessarily. The persistent module is definitely a consistent option. Kafka and Akka are really just different animals with different goals. Kafka is effectively pub/sub, Akka essentially takes a much more event sourced approach. I've worked systems that use both, but they use them for very different purposes.