12

I've studied some CQRS sample implementations (Java / .Net) which use event sourcing as the event store and a simple (No)SQL stores as the 'report store'.

Looks all good, but I seem to be missing something in all sample implementations.

How to handle the addition of new report stores / screens, after an application has gone in production? and how to import existing (latest) data from the event store to the new report store?

Ie:

Imagine a basic DDD/CQRS driven CRM application. Every screen (view really) has it's own structured report store (a SQL table). All these views get updated using handlers listening to the domain events (CustomerCreated / CustomerHasMoved, etc).

One feature of the CRM is that it can log phone calls (PhoneCallLogged event). Due to time constraints we only implemented the logging of phone calls in V1 of the CRM (viewing and reporting of who handled which phone call will be implemented in V2)

After a time running in production, we want to implement the 'reporting' of logged phone calls per customer and sales representative.

So we need to add some screens (views) and the supporting report tables (in the report store) and fill it with the data already collected in the Event Store...

That is where I get stuck while looking at the samples I studied. They don't handle the import of existing (history) data from the event store to a (new) report store.

All samples of the EventRepository (DomainRepository) only have a method 'GetById' and 'Add', they don't support getting ALL aggregate roots in once to fill a new report table.

Without this initial data import, the new screens are only updated for newly occurred events. Not for the phone calls already logged (because there was no report listener for the PhoneCallLogged event)

Any suggestions, recommendations ?

Thanks in advance,

Remco

Remco Ros
  • 1,467
  • 15
  • 31

3 Answers3

8

You re-run the handler on the existing event log (eg you play the old events through the new event handler)

Consider you example ... you have a ton of PhoneCallLoggedEvents in your event log. Take your new Handles and play all the old events through it. It is then like it has always been running and will just continue to process any new events that arrive.

Cheers,

Greg

Greg Young
  • 2,513
  • 1
  • 21
  • 13
  • So basically you'll need to integrate some kind of one-time event runner (report import) in your deployment process? – Remco Ros Apr 14 '10 at 14:24
  • And you'll need 'Query' access to the Event Store: store.GetAllEvents(); This seems counter intuitive because (as i said) All implementations of an EventStore I have seen (both .net / java) don't allow such read access... – Remco Ros Apr 14 '10 at 14:31
  • 1
    Usually the provided implementations only contain what is needed at runtime, but not what is needed for maintenance. – thinkbeforecoding Apr 14 '10 at 14:44
  • Very clear. So it's all about separation. Still you need some mechanism to determine which events to send trough this 'maintenance' component. You don't want to miss events or handle the same events twice because of the time frame the new handler subscribes for new events and the initial import of historic events. – Remco Ros Apr 14 '10 at 14:57
  • Generally what is best is to: Subscribe to events of that type. Query for previous events (there may be multiple here) let subscriptions queue and process previous events one by one. Then eat off subscribed events (must be idempotent so you don't double process) Part of the reason why the repository does not support it is that you would never want to do this *in* the domain. The repository is the contract that the domain has with the event storage. – Greg Young Apr 14 '10 at 15:27
  • It will take some time to fully implement a workable production ready, deployable solution. But at least I learned some more about the deeper concepts. Thanks again. – Remco Ros Apr 14 '10 at 16:12
  • But how would you do this if you didn't implement event sourcing with your cqrs solution? – Sinaesthetic Nov 29 '16 at 17:23
2

For example in Axon Framework, this can be done via:

JdbcEventStore eventStore = ...;

ReplayingCluster replayingCluster = new ReplayingCluster(
            new SimpleCluster("replaying"),
            eventStore,
            new NoTransactionManager(),
            0,
            new BackloggingIncomingMessageHandler());

replayingCluster.startReplay();

Event replay is an area that is not completely documented and lacks mature tooling, but here are some starting points:

AlexO
  • 1,311
  • 12
  • 18
1

The 'EventRepository' only contains these methods because you only need them in production.

When adding a new denormalization for reporting, you can send all event from start to you handler.

You can do this on your development site this way :

  • Load your event log to the dev site
  • Send all events to your denormalization handler
  • Move your new view + handler to your production site
  • Run events that happened inbetween
  • Now you're ready
thinkbeforecoding
  • 6,668
  • 1
  • 29
  • 31
  • This is not really doable in the way you describe. We don't have access to production data (events). But based on your and Greg's answer, I see now that you'll need some kind of one-time import component (Event Runner) which is run only once when deploying the new handler to production. – Remco Ros Apr 14 '10 at 14:28
  • What prevent you from getting the events ? Once the happened, they'll never change, so there is no risk to make a copy. But anyway, you'll need a component (or process) that reads all events and call the handler. – thinkbeforecoding Apr 14 '10 at 14:42