Event Sourcing with Kafka for Microservices

Question

I have several microservices that are using data from kafka. They consume and produce data to the broker.

Those microservices only have a volatile storage (hazelcast). When the storage gets lost I need to reconstruct it based on the master data sitting in kafka.

My naiv implementation just consumes those data again, but then I produce some old data to the broker. This again triggers the other microservices which seems to be a bad idea.

Is there a standard way to handle this usecase? For me it seems that this is a very common problem, or do i get something wrong?

Reconsuming a topic is what Kafka Streams would do in the event of an app restart in order to recreate a in memory KTable — OneCricketeer, Oct 05 '18 at 14:01
The problem is not the reconsumption of events. What I want to do is to prevent the microservice from producing events based on old data — rloeffel, Oct 05 '18 at 15:02
Doesn't that depend on how you are generating the data to begin with and your definition of old? — OneCricketeer, Oct 05 '18 at 17:36
By "old" i mean already consumed data... I would like to prevent my service from republishing events due to the recovery of my local storage. For me this seems like a natural requirement... — rloeffel, Oct 06 '18 at 11:20
A consumer group keeps track of the data it's already consumed and the offsets for which that data came from... If you are not producing new data and your consumers committed the offsets for data they've read, then nothing should be reconsumed. As far as producers are concerned, you must find a way to mark yourself what has already been produced in order to prevent duplicates — OneCricketeer, Oct 06 '18 at 16:45
I think I found now a way how I can implement this. I'm just astonished that there is no out-of-the-box feature supporting this requirement. Im going to post my solution on monday.... — rloeffel, Oct 06 '18 at 20:18

score 0 · Answer 1 · answered Oct 05 '18 at 17:00

0

This has been asked before.

It should not matter that you are using Kafka as your event store since the problem is the microservices re-sending the events.

answered Oct 05 '18 at 17:00

satm12

60
1
6

Agree. But what I dont like in the proposed solution is that the other microservices need to handle this shortcomming. I do not want to publish events that I already published before. Just because i need to restore my local datastore i do not want to disturb other services... – rloeffel Oct 06 '18 at 11:14

score 0 · Answer 2 · edited Oct 09 '18 at 04:19

After spending several days i came up with the following solution.

The key idea is to do the synchronization in two modes, namely Recovery and Normal

In the recovery mode I only consume data but I do not produce any data.
In normal Mode I consume and produce data.

In Kafka I implemented this using two listeners belonging to different consumer-groups. On startup all listeners are stopped and I decide with kind of listener gets enabled. Once the offset of all recovery listeners reaches the watermarks of the normal listeners I stop the recovery listners and start the normal listeners.

Below the relevant part of my code:

public void startListeners() {
    log.debug("get partitions from application");
    final List<KafkaPartitionStateKey> partitions = getPartitions();

    log.debug("load partition state from hazelcast");
    final Map<KafkaPartitionStateKey, KafkaPartitionState> kafkaPartitionStates = kafkaPartitionStateService.loadKafkaPartitionStateMap();

    log.debug("check if in sync");
    if (areAllPartitionsReady(partitions, kafkaPartitionStates)) {
        log.info("all partitions ready, not need to start recovery");
        this.messageListenerContainers.forEach(this::startContainer);
        return;
    }

    log.debug("load consumer group offsets from kafka");
    consumerGroupOffsets = getConsumerGroupOffsets();

    log.debug("create missing partition states");
    final List<KafkaPartitionState> updatedPartitionStates = getOrCreatePartitionStates(partitions, kafkaPartitionStates, consumerGroupOffsets);

    log.debug("check if all partitions are ready");
    if (getNumberOfNotReadyPartitions(updatedPartitionStates) == 0) {
        log.info("all partitions ready, no need to start recovery");
        this.messageListenerContainers.forEach(this::startContainer);
        return;
    }

    log.info("----- STARTING RECOVERY -----");
    this.recoveryListenerContainers.forEach(this::startContainer);
}

I hope this is usful for somebody...

Event Sourcing with Kafka for Microservices

2 Answers2