1

Say I am using Kafka as the event-driven backbone for all my microservices in my system design. Many microservices use the events data to populate their internal databases. Now there is a requirement where I need to create a new service and it uses some events data. The service will only be able to consume events after the time it comes live and hence, won't have a lot of data that it missed. I want a strategy such that I don't have to backfill my internal databases by writing out scripts.

What are some cool strategies I can have which do not create a huge load on Kafka & does not account for a lot of scripting to backfill data in the new services that I ever create?

Gitesh Khanna
  • 130
  • 9
  • Interesting problem! My first reaction is that you would need a persistent event log so that services can seed themselves without coupling to the interfaces of others, but obviously that’s difficult if it didn’t exist from the start. Do you have a good example of a service that would need to populate itself with historical event data? – David T. May 09 '20 at 20:59
  • Say I am creating something like amazon where whenever there is a new seller onboarded. There is some seller metadata which multiple services use, but store it in their internal database in different ways(for their own optimal way of fetching). The events will then be somewhat like seller_metadata_create, seller_metadata_update. So basically I'll need that if a new service starts listening to these topics. It should have all the prior knowledge of everything that was ever sent to this topic. – Gitesh Khanna May 12 '20 at 07:32

1 Answers1

2

There are a few strategies you can have here, depending on how you publish data to a kafka topic. Here are a few ideas:

  • first, you can set the retention of a kafka topic to be forever, meaning that it will store all the data. This is OK as kafka is built for this purpose as well. See this. By doing this, any new service that come alive can start consuming data from the start.

  • if you are using kafka for latest state publishing for a given entity/aggregate, you can also consider configuring the topic to be a compacted. This will let you store at least the latest state of your entity/aggregate on the topic, and new consumers that starts listening on the topic will have less data to configure. However, your consumers still need to know how to process multiple messages per entity/aggregate as you cannot guarantee it will have exactly one message in the topic.

tugberk
  • 57,477
  • 67
  • 243
  • 335
  • The 1st point will probably result in blocking my kafka queue, if the events data are too many. Although the 2nd point seems to be a very interesting solution. Will go through it in detail and update if it solved my problem! Thanks! – Gitesh Khanna May 12 '20 at 07:35