I am building a pipeline to import table data from one database to another using apache-kafka and apache-kafka-connect. I can't use log-based CDC (and such great things like debezium), so I have to use query-based CDC and JDBC Source Connector.
JDBC Sink Connector is used to import from Kafka to the target database. This connector supports deletes:
A record with null value is considered a tombstone event and result in deleting the corresponding row in the destination table.
So if I were to use Debezium, I could use the tombstones.on.delete option to emit tombstones, however, in the case of the query-based CDC approach, I need to use a special column to mark deleted rows.
Is there any solution for transformation messages (representing "soft deletes") into tombstone events (or, in other words, to set value of such messages to null)? Maybe I could use some kind of conditional Kafka Connect SMT or something else?
Overall, is it a good idea to turn "soft" deletes into real ones?
Thanks in advance!
Similar questions:
