12

I am playing around with the Kafka Connect JDBC connector and specifically looking at what the actual format of the data that is put onto the topic is.

I have been able to see new inserts and updates to the database, but I have not been able to detect deletes from the database.

First: Does the JDBC source support detecting these changes? I can't find documentation one way or another.

If it does, what format does it take on the actual topic?

ebensing
  • 6,409
  • 4
  • 18
  • 20

1 Answers1

26

The Confluent JDBC source connector is able to capture "soft deletes", where the "deleted" rows are simply marked as such by your application but are not actually removed from the table. Since the rows are still there, the connector can see their changes. However, the connector is not able to capture rows that are deleted from a table, since the connector queries the source tables via JDBC and thus is unable to see rows that are removed from the tables.

Other connectors are able to capture all of the changes by tapping into the database transaction logs or write ahead logs, using techniques known as change data capture, or CDC. Each DBMS is different, and therefore requires a connector written specifically for that DBMS. For example, the Debezium project has Kafka Connect connectors for MySQL, PostgreSQL, and MongoDB, and is working on connectors for Oracle and SQL Server.

Randall Hauch
  • 7,069
  • 31
  • 28
  • 1
    Perfect, this is exactly what I was looking for. Thanks! – ebensing Jul 19 '17 at 17:37
  • if my SQL Server version is old (2012) and does not have the CDC feature, what would your suggestion be? Upgrade the DB to a later version and enable CDC and work with Debezium SQL Server Kafka Connect connector, or use the Confluent JDBC source connector? – cryanbhu Feb 08 '22 at 09:34