1

I'm streaming database change events from MySql database to Kafka using Debezium MySql connector. I need to apply specific transformations to records from some specified tables (but not from the others). Is there a way of doing it with using only Kafka Connect Single Message Transforms (without Kafka Streams or KSQL)? I do not even mind writing a custom SMT. But I cannot find how can I specify to messages from which tables/topics to apply SMT. Thank you.

[UPD] After brief investigation I decided to use KSQL for that. Thanks everyone for answers

Ruslan
  • 11
  • 2
  • You can create different connector for each table and then apply SMT to only needed topics – Bartosz Wardziński Apr 11 '19 at 13:45
  • Yes, that is also an option, thanks. But I want someone to confirm that applying SMT to specific topics only is impossible before I consider any other scenario.. – Ruslan Apr 11 '19 at 13:50

2 Answers2

1

You can't apply SMT only to particular topic. Connector whether it is Sink or Source always apply defined SMT to each message, that it processed.

There is an option that SMT have some complex logic and depends on some data inside the message (topic name partition number, key, etc) do some work or not. Describe approach is not recommended, because SMT should make lightweight processing (for more complex you should use Kafka Streams). That is the reason why SMT from org.apache.kafka.connect.transforms.* don't have such logic.

If you want to apply SMT to only specific topic, you can create different connector for each table (topic).

Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30
1

If you are implementing a custom SMT, you can simply examine the topic of incoming records and just return records un-altered for those topics you don't wish to apply your transformation logic.

Gunnar
  • 18,095
  • 1
  • 53
  • 73
  • Yes, that is the way to go in case of implementing a custom SMT. I was asking if there is a way to do it without writing my own SMT. Eventually I decided to utilize KSQL for this task. – Ruslan May 03 '19 at 09:24