4

My question is related to Change data capture and real time updates. I have a mysql database which has close to 250 tables whose change data is captured using debezium and sent to kafka on their respective topics.

These tables are highly normalised and I am interested in a denormalised view which is created by joining some of the tables. Now I need to know whenever there is a change in the table, I get to see that as a change in the denormalised topic in a near real time.

How can I do so. How does log compaction works? Do I need to log compact all the topics in order to get denormalised topic view?

What other possibilities I have here?

clinton3141
  • 4,751
  • 3
  • 33
  • 46
Nipun
  • 4,119
  • 5
  • 47
  • 83

1 Answers1

2

This is not very straight forward to accomplish.

The following are some options you could try.

  1. The first option is using KSQL

https://debezium.io/blog/2018/05/24/querying-debezium-change-data-eEvents-with-ksql/

  1. The other option is to create aggregate views

https://debezium.io/blog/2018/09/20/materializing-aggregate-views-with-hibernate-and-debezium/

  1. Another option that we tried for a similar problem was to have streaming listener on Kafka and then have logic there to merge data into de-normal data structures that have been set up on another datastore
Ramachandran.A.G
  • 4,788
  • 1
  • 12
  • 24