0

What is the best way to keep a view table up to date, using the CQRS/Event Sourcing pattern?

The most common way is through KSQL queries, but I would like to know if there is a more generic way, for example, using a relational database.

enter image description here

The more time your app is running, you will have to read more data, making this unsustainable. How do you deal with this? Do you just read all the events and compute them every time one new command is executed?

juanmorschrott
  • 573
  • 5
  • 25
  • Wouldn't you just update the view table by reading the events as they are generated? Using some kind of streaming framework like Kafka Streams or Flink – Gerard Garcia Jun 08 '22 at 12:25
  • Yeah, you are right. You have the option of using KSQL, but I want to know if there is another way. All the examples I found were about Axon, Kafka or custom databases. – juanmorschrott Jun 08 '22 at 14:36

1 Answers1

1

The typical approach to this is to store as part of the view's state where it is in the event stream (i.e. an offset). For example, your view can store that it's current up through event 1003 for some partition (e.g. an entity ID) in the event stream. When querying for events in that partition, the process updating the view table asks for events after 1003. One can trigger that process periodically, or have it run continuously, or signalled as part of the process of writing commands.

One of the views can be "publish these events to Kafka"; in which case other views can use the consumer group/offset commit functionality to process the events at-least-but-typically-not-more-than-once.

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
  • Ok, so when you store the command, the view is updated just with a slice of the data (events log), is that what you mean? – juanmorschrott Jun 08 '22 at 14:41
  • And if you're talking about processing commands (i.e. on the write-side), one can snapshot, which is essentially saying "as of this event, the state was". Then the process of recovering state is "find the latest snapshot, query for events after that snapshot, apply those events to the snapshot". – Levi Ramsey Jun 08 '22 at 14:43
  • The view doesn't care about commands (except in the trivial perspective that one can view an event from the write-side as a command to the read-side) – Levi Ramsey Jun 08 '22 at 14:44
  • Ok, thank you very much sir. I was curious in the view table creation. Because reading all the logs, every time a command is stored in order to create such table view is not an option. – juanmorschrott Jun 08 '22 at 14:45
  • You're generally not storing the command when event sourcing (that's command sourcing: there's a subtle difference there) – Levi Ramsey Jun 08 '22 at 14:45
  • 1
    For very ad-hoc and non-interactive queries, playing back the events is an option. Otherwise you typically have a database serving as the read-model, which gets updated incrementally based on the events persisted as the write-model. – Levi Ramsey Jun 08 '22 at 14:48
  • Having that database for reads is technically optional: it's just an optimization. – Levi Ramsey Jun 08 '22 at 14:49
  • OK thank you very much. I'm going to sort these ideas out in my head. I changed the image to make the question more clear. – juanmorschrott Jun 08 '22 at 14:55