CQRS / Event Sourcing / Event bus / Timing

Question

I have a case of timing in my CQRS/ES design. For the sake of discussion let's base it on Microsoft's example on this topic, conference management (https://msdn.microsoft.com/en-us/library/jj554200.aspx).

Two contexts: conference management and order management. In conference management you create conference and change its properties (e.g.: maximum seats). Conference management communicates with order management through event bus. This way order management knows when a new conference is created, and instantiate a tracking object (seat availability) within its context. Event like "max seat availability is reduced from 100 to 80" is also communicated from conf mgmt to order mgmt through event bus.

• Let's say at minute 1 conference is created (maximum seats 20).
• At minute 4 the event arrives at order mgmt context, and hence a seat availibility is created.
• At minute 7 user made an order (through order management), buying the entire 20 seats. (this should be communicated as well from - order management to conf management so that conf admin can make proper decision).
• At minute 8 conf admin made change to seats availability (in the conf mgmt context), reducing it to 15.
• The notification from order mgmt (about the fact that all the 20 seats have been sold) arrives at minute 9.
• What to do now? (at the conf mgmt? you weren't supposed to reduced the seats to 15 when you already have received payment for 20; but at minute 8 the admin didn't know about it).

That's the first case.

Now the second case:

• Let's say at minute 1 conference is created (maximum seats 20).
• At minute 4 the event arrives at order mgmt context, and hence a seat availibility is created.
• At minute 6 conf admin made change to seats availability (in the conf mgmt context), reducing it to 15.
• At minute 7 user made an order (through order management), buying the entire 20 seats. He doesn't know that the availability has been reduced to 15 because the event hasn't arrived at this point.
• At minute 10 the notification from conf mgmt finally arrived.
• What to do now?

These problems stem from technical issues (latency in event transmission). Is there a purely technical ways to avoid it? If not, what would be the business-way / design-way to overcome it?

Added note: I'm aware this is the eventual-consistency problem pertinent to CQRS/ES architecture. Within a single context, this might be easier to tackle, because you can have commands (e.g.: undo) for that. But, between bounded contexts you shouldn't pass on commands, you only communicates with events, and I think event is not the right abstraction for this (because it represents something that happened). Or am I missing something?

Added note: maybe this article can provide more context and hint to solution, though for this specific case in this question I don't think so (it's not about out-of-order message). http://blog.jonathanoliver.com/cqrs-sagas-with-event-sourcing-part-i-of-ii/

Added note: or maybe..., in the order management context, we can simply queue the command for buying a seat, until we know that we have received the latest / last event from the conference management context. I mean, the events come with a timestamp, right? So we can compare the timestamp of the last event we received from conf mgmt context with the timestamp of the command for seat purchase. If the ts of last event is less than the ts of the command, we simply hold off the execution of the command, until we receive an event whose ts is bigger than ts of the command. That kind of thing would be the responsibility of the process manager (saga).

Would that work? Is it the correct approach?

Added note: relevant thread > Implementing a Saga/Process Manager in a CQRS http application . I think we're on the right track with process manager. As the answerer said: "You simply don't answer "Order Confirmed" immediately. Take a look at how Amazon and other shopping sites do it: Upon Order submission, you just receive an "Order Accepted confirmation (e.g., HTTP Code 202 Accepted).".

It's up to your process-manager's logic to decide when to actually execute the command (based on certain condition, internal state maybe, or in this case the latest received event from the other context).

Any opinions?

Thanks, Raka

Answer 1

Three possibilities

First, you accept the eventual consistency. As noted in the comment, in many eventual consistency scenarios it turns out that the business will have a way to mitigate the problem if made aware of it in a prompt fashion - so you would need something watching the events coming out of your model that notices that you have a discrepancy between the number of seats reserved and the number of seats sold, and perhaps a task based UI to allow the human being to do the right thing [tm].

Second, check with the business to see if you are actually modeling the real business process correctly. You might, for instance, be missing a couple stages in the purchasing of the seats; order management may need to reserve the seats before confirming the order.

(Which is to say, the event that says a customer wants 20 seats may be a separate thing from the event that says we actually sold them to her, both of which are distinct from the event that conference management has reserved the seats for that customer).

Note that this doesn't actually save you from the business problem - you still have a race condition between a customer wanting to buy 20 seats and a conference organizer wanting to reduce the size of the audience. What you do get is a slightly clearer failure mode (if the client wins the race to claim the seats, then conference management can refuse to reduce the count. Likewise if the organizer wins the race, the attempt to reserve the seats is declined).

This assumes that conference management should maintain an invariant that the number of reserved seats is less than the number allocated. That may not actually be true in your business. Also, it may be useful to the business to be able to track the number of times a customer tried to reserve more seats than were available.

Third, it may be that your service boundaries are drawn in the wrong place. Udi Dahan wrote

Any piece of data or rule must be owned by only one service.

If seats in conference management and seats in order management are the same thing, then maybe they belong together (same service boundary).

My recommendation, based on the principle that the automation should be able to get out of the way of the domain experts when the need arises, is that you should be thinking primarily in terms of the first approach - can you surface issue and surrender control of it to the business experts. If you get that much right, you are likely to have a successful project.

Answer 2

As mentioned in my comment and the other answer here, you should first of all check with the business what happens in this scenario. Often things are much simpler if you put the question to the business.

The usual way to deal with these things is issue some sort of compensatory action. For example, if you were to place an order on a standard ecommerce site but by the time it is processed the goods are out of stock, you would normally receive a compensatory action - an email saying it is out of stock, maybe an offer on an alternative item etc...

In you example, if someone placed an order for 20 seats but they were subsequently reduced to 15, you could:

• cancel the order completely, offering a full refund
• reduce the order to 15 and offer a refund for the remaining seats
• keep the order as is, accepting the overbooking - maybe there are always people who cancel or don't turn up

They're just off the top of my head...

The important thing is to ask the business. They'll often surprise you!

If this answer comes back that it must never happen, then the other answer has some sensible suggestions on this.

Answer 3

Lets kick off with, what I see as, the biggest problem in your design: Your aggregate takes a long time to update.

A nice way to design your product is something like this: GUI -> API -> Command -> CommandHandler[Tell aggregate something can be updated] --> Event [First goes to aggregate, then to other subscribers]. You also want to make sure that Events are applied before a new Command gets handled. So your CommandHandler should finish proccessing a Command before starting on a new one. Now, given how many Commands you expect to processes for any given time unit, you will have to model it thereafter. But if you do this you make sure that the Aggregate can make the appropriate decision if the Command is a valid one or not.

Why do I deem this as a big problem? Because if you have a Command that wants to book all seats, you want to be sure that there are that many seats available. You do not want an Event lagging behind that reduced the number of available seats. The aggregate has to have the real information, always. Otherwise you will be creating non-valid Events, which is a big no-no.

All of this bring up the question "Where should the business logic be?" Well, that can be somewhat tricky. The aggregate is the only (or should be the only) one that knows which state it is in. But the CommandHandler should also check if the Command is a valid one. So your logic can become split between the aggregate and the commandHandler. A simple way to do the split is say that the CommandHandler should only check if it's a valid command, while the Aggregate should check 'can I really do this?'.

So, if we imagine the above mentioned split and we have your Scenario 1.

1. CreateConferenceCommand("MyConference", 20 seats)
2. ConferenceCommandHandler(CreateConferenceCommand command)
   • Check if command contains all needed info
   • Tell aggregate a command arrived and asked to update with the info.
3. Aggregate OKs the changes.
   • In this case the aggregate was created and has 20 seats.
4. ConferenceUpdatedEvent created and published.
5. OrderCommand("MyConference", 20 seats)
   • What this actually means (if it has been confirmed and all that) is up to you.
6. ConferenceCommandHandler(OrderCommand commmand)
   • Check if command contains all needed info
   • Tell aggregate a command arrived and asked to update with the info.
7. Aggregate OKs the changes.
   • It had 20 seats, so it OKs the change.
8. ConferenceUpdatedEvent created and published
   • "MyConference" reduces the number of available seats with 20.
9. ChangeSeatsCommand("MyConference", 15 seats)
10. ConferenceCommandHandler(ChangeSeatsCommand command)
    • Check if command contains all needed info.
    • Tell aggregate a command arrived and asked to update with the info.
11. Aggreget does not OK the changes.
    • Since all seats have been booked, and we assume they have been paid for, we can't simply change the number of seats. So we block the change.
12. Inform user of Error.

You have by this made sure you can't reduce the number of seats when they are all booked. But since you have to be able to actually change them, as is the case with our (stupid) reality because of eg. fires, you will have to make some mechanism for this as well. Personally, I would probably make a new Command+Event for it that involves some repayment to the customer and some apologies.

I hope this somewhat helps you.