0

I'm trying to create a custom transform for Kafka CDC.I want running a replication between MySQL and MongoDB. I prefer when a row update/insert in MySQL database the records the related to that row (A hierarchy tree structure) corresponding complex JSON document(a hierarchy JSON document) update/insert in MongoDB. Can I implement custom transform for Debezium?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Mohammad Hossin Samadian
  • 107
  • 11
  • You may refer to the following two questions. https://stackoverflow.com/questions/57604283/write-a-custom-kafka-connect-single-message-transform https://stackoverflow.com/questions/64250997/how-to-write-custom-smt-in-kafka-source-connector-to-obfuscate-private-data – C.C. Hsu May 15 '22 at 06:32

1 Answers1

1

A custom transform for Debezium is no different from a custom SMT for Connect framework, ignoring the defined scehma of the Debezium event.

  1. Create a Java project with a dependency on at least org.apache.kafka: connect-transforms
  2. Implement the Transformation interface.
    • Ideally, add unit tests for this as well
    • You can look into the Kafka source code for how existing ones are implemented.
  3. Package it as a JAR. If it needs external dependencies, a shaded JAR will work as long as you relocate any packages that Kafka itself will have on its classpath. For any of those, you should ensure provided scope; for example, kafka-clients or jackson-related libraries.
  4. Copy the JAR into a directory listed on the plugin.path of the connect worker(s) configs.

https://docs.confluent.io/platform/current/connect/transforms/custom.html

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245