2

I'm experimenting with event sourcing / cqrs pattern using serverless architecture in Azure.

I've chosen Cosmos DB document database for Event Store and Azure Event Grid for dispachting events to denormalizers.

How do I achieve that events are reliably delivered to Event Grid exactly once, when the event is stored in Cosmos DB? I mean, if delivery to Event Grid fails, it shouldn't be stored in the Event Store, should it?

Liero
  • 25,216
  • 29
  • 151
  • 297

2 Answers2

11

Look into Cosmos Db Change Feed. Built in event raiser/queue for each change in db. You can register one or many listeners/handlers. I.e. Azure functions.

This might be exactly what you are asking for.

Some suggest you can go directly to cosmos db and attach eventgrid at the backside of changefeed.

HakonIngvaldsen
  • 354
  • 2
  • 12
3

You cannot but you shouldn't do it anyway. Maybe there are some very complicated methods using distributed transactions but they are not scalable. You cannot atomically store and publish events because you are writing to two different persistences, with different transactional boundaries. You can have a synchronous CQRS monolith, but only if you are using the same technology for the events persistence and readmodels persistence.

In CQRS the application is split in Write/Command and Read/Query sides (this long video may help). You are trying to unify the two parts into a single one, a downgrade if you will. Instead you should treat them separately, with different models (see Domain driven design).

The Write side should not depend on the outcome from the Read side. This means, that after the Event store persist the events, the Write side is done. Also, the Write side should contain all the data it needs to do its job, the emitting of events based on the business rules.

If you have different technologies in the Write and Read part then your Read side should be decoupled from the Write side, that is, it should run in a separate thread/process.

One way to do this is to have a thread/process that listens to appends to the Event store, fetch new events then publish them to the Event Grid. If this process fails or is restarted, it should resume from where it left off. I don't know if CosmosDB supports this but MongoDB (also a document database) has the rslog that you can tail to get the new events, in a few milliseconds.

Constantin Galbenu
  • 16,951
  • 3
  • 38
  • 54
  • Nice explanantion, dont know if it is a tech answer but at least it gives something to think about ;-) – Thomas Oct 17 '18 at 06:25
  • @Thomas thanks. I think that the OP has a bigger problem: the design of the Application and maybe he should see CQRS in a different, deeper way. – Constantin Galbenu Oct 17 '18 at 06:27
  • Yeah it does not make sense based on the question – Thomas Oct 17 '18 at 06:29
  • @ConstantinGalbenu: I understand that I command stack should not depend on query stack, but I was talking only about delivering events to underlying messaging system. It should not happen that event is stored but not dispatched. I guess that EventStore database from Greg Young solves this out of the box, but I want cloud native solution – Liero Oct 17 '18 at 08:12
  • @Liero the command stack ends immediately after the events get persisted. How the events get to the Readmodels is not its business. – Constantin Galbenu Oct 17 '18 at 08:14
  • Probably not command stack problem concern, but it is a concern of overall CQRS architecture. I was just hoping that there is some reliable built in mechanism, in CosmosDB resp EventGrid. – Liero Oct 17 '18 at 08:50
  • @Liero not that I know of. These are the event sources that work with EventGrid: https://learn.microsoft.com/en-us/azure/event-grid/event-sources – Constantin Galbenu Oct 17 '18 at 09:07
  • In that case I think that the Event Handlers should pull events somehow (rather that rely on events being pushed into them) in order to ensure that every event is processed, right? EDIT: Oh, I noticed your last paragraph only now. It answers it perfectly – Liero Oct 17 '18 at 11:53
  • @Liero, Azure function has cosmos db trigger, this may help ? https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-cosmos-db-triggered-function – Thomas Oct 18 '18 at 00:07