4

I'm looking for options for our eventsourcing solution. I received recommendations to use Eventbridge Archive since we are already using AWS Eventbridge.

For what I found about the Archive feature, it can store all the events that we send to the eventbus and can replay them later based on a filter.

The problem is that I couldn't find a way to read data from this Archive without replaying the events.

Does anyone know if there is an API that allow me to do that?

Everything I can find on Google is about create, archive and replay, but nothing about reading data from events stored in the Archive.

I found this other question but it didn't have any answers.

Raul
  • 167
  • 9
  • To clarify, you're looking for a means to pull out one specific event? – Levi Ramsey Jan 06 '22 at 13:15
  • Basically, the EventBridge docs tell you on its front page that it's an event bus, designed to support event-driven systems. It's not an event store, it cannot be used as a database for storing events. I don't write it as an answer as it's not answering your question, but I can suggest you drop that advice. – Alexey Zimarev Jan 29 '22 at 20:04
  • @AlexeyZimarev, I understand that Eventbridge does not support that. I was talking about the archive. Eventbridge has this archive feature that stores events based on a filter like an Eventbridge rule. But it seems like the only way to access those archived events is to use the replay feature, which is not ideal. – Raul Jan 31 '22 at 14:14
  • @LeviRamsey not exactly. My objective is to store all the events in a way that I can run queries on them. I found this Archive feature but it looks like it does not allow queries. I think a way to do what a want is to send all events to a Lambda and then write them on a database. – Raul Jan 31 '22 at 14:16
  • @Raul what you said is what I mean by "not fit for the job". You need a database, not a broker. – Alexey Zimarev Jan 31 '22 at 20:07

1 Answers1

1

(Disclaimer: this answer is informed only by a quick skim of the Eventbridge docs)

Event sourcing is a bit of an overloaded term: it's been used to mean everything from an architecture built on exchanging events to using persisted events as the source of truth.

In the latter usage of the term (which might be called "going full event sourcing"), strictly speaking, all reads are replays of an event stream, with there being multiple event streams and the same event often being written to multiple event streams (e.g. a "cart checked out" event may well be written to an event stream for that particular cart and to an event stream consisting only of "cart checked out" events).

EventBridge and EventBridge Archive would appear to be sufficient to "go full event sourcing": each one of these event streams becomes an event bus. It might not be ergonomic to create new buses on the fly and the mechanism for publishing the same event to multiple streams may be complex (e.g. a service getting fed base events and continuously projecting them to other streams), assuming that you will have somewhat less than 100 entities (as AWS accounts are limited to 100 event buses). If you have more than 100 entities, then EventBridge isn't a fit for event sourcing.

That said, in event sourced systems it's almost universally common to have some read functionality which doesn't map well to replaying a stream (e.g. adhoc queries/aggregations over the events). This is where Command Query Responsibility Segregation comes in: it's OK if adopting CQRS to

  • have multiple data models in the application
  • have workloads use the data model which is the most appropriate for that workload

The command-processing/write side of an application often benefits from event sourcing (especially if there's infrastructure to help with some combination of concurrency control/caching/maintaining single-writer). You then project event streams into other data models (e.g. databases or search systems or alerting infrastructure) which can be tuned for the queries that will be performed against them.

So your question is perhaps better expressed as "how do I do CQRS in an event sourced system", and thankfully, there's a well-established body of technique for that. In your case, sending all the events in (at least some of) the streams to a Lambda which writes them to a database is likely to be a reasonable way to support CQRS.

Levi Ramsey
  • 18,884
  • 1
  • 16
  • 30
  • 2
    The issue with "event sourcing" is a term is not that is "overloaded", but that it's misunderstood and bent by certain vendors to fulfil their marketing purposes, making their product fit for what it doesn't. Event Sourcing as defined by Greg Young more than a decade ago is just a way to persist the state of an entity by storing its state transitions as events. This implies that restoring the state of a given entity requires reading all those events for a single entity. Message brokers are simply not designed for that, none of the message brokers that exist today fit that basic requirement. – Alexey Zimarev Jan 31 '22 at 20:06
  • And if you have fewer than 100 entities, EventBridge plus Archive satisfies that requirement. Within that quota, you can read all the events for a single entity. – Levi Ramsey Jan 31 '22 at 23:19
  • I'll note, @AlexeyZimarev, that you work for a vendor of an event sourcing solution. – Levi Ramsey Jan 31 '22 at 23:26