How to handle many instances of read model producers in event sourcing/cqrs?

Question

I'm often using kubernetes for my deployments. I don't see how building read model could be done when we want to have multiple read model producers. If I spin up a new service that needs to rebuild its read model I would subscribe to event store and replay all events since the beginning. Then when service is up to speed I would just listen for incoming new events. Everything looks fine when we have single instance of this service but if for example we have 2 instances then they both would receive events and try to apply it twice. After some searching most common solution I found is to use only single subscriber/instance for given read model database. This approach from my perspective have a single point of failure. If the subscriber fails for some reason but don't crash immediately then kubernetes will not spin new instance of this service. How would I handle such case?

Currently I see it like this: CommandService(multiple instances) => EventStore => ReadModelProducerService(single instance) => ReadModel <=> QueryService(multiple instances). If this single instance of ReadModelProducerService that is generating read model fails then the app is basicaly down.

Answer 1

There are at least three issues of concurrent subscribers that execute the same projection code.

1. Double load on the target database as they will update the same records or documents
2. If the projection code isn't idempotent (it should be, but still), it will quickly go to hell
3. They will override each other's checkpoint (which might be the least problematic)

I'd say that the first problem is the most obvious, and it may lead to some undesired consequences like record locks and timeouts.

I wouldn't be that concerned about a "single point of failure" as there are many other "single points". Event store itself, Kubernetes, the projection sink - all these components can fail.

If you trust Kubernetes more than your own code, you can avoid the situation when the subscription has crashed but stays alive in "zombie mode". The problem is well-known, as well as the solution. You need to add a histogram metric to the subscription to measure the processing rate. Another useful metric is the subscription gap, which will grow if the subscription is slow or stopped. In many cases, you can detect the subscription drop as it gives your application a signal (can't connect to the db, etc), which can be used as a healthcheck, forcing Kubernetes to restart the pod. I wrote about these things in Eventuous docs.

Answer 2

The quick answer is that It Depends, specifically on what your read model's requirements are.

It's absolutely possible that your read model requires processing every event synchronously (i.e. for any 2 events, there is exactly one valid ordering in which those events can be applied): if this is the case, then yeah, you can only have one process (and that process basically has to be single-threaded, too). Your throughput will be limited. "Them's the breaks," unless you can refactor your model of events or negotiate a requirement change to loosen this requirement.

If this isn't the case, you can typically partition/shard the events in some way: the mechanism for this will vary by event store (e.g. in Akka Persistence you might shard by persistence (typically analogous to an aggregate) ID, or you might tag the events), but the key thing will be to have events where the order in which they're applied matter be in the same partition/shard. Each instance can then take responsibility for different partitions/shards: this will typically entail some form of coordination between the instances, like using a strongly consistent data store (e.g. etcd in Kubernetes) for leases. You'll also want to dig into the ordering guarantees provided by the event store: for instance, events for a given aggregate root will have a defined order but ordered events by aggregate root might not be interleaved in any predictable order.

Including timestamps, sequence numbers, or vector clocks in the events themselves can also facilitate reconstructing a global ordering after the fact, though this does often entail some explicit modeling of uncertainty with respect to time (e.g. modeling retractions and confidence).