5

I'm using the Kafka connect JDBC source connector to read from a view in a database and post it on kafka, it is working fine.

My use case is that a user can create multiple objects and the order of the objects is important in my application. I would like to use the user id as the message key of all the messages I'm posting into the topic to maintain their order.

My question is how can I define a message key in the Kafka connect source connector?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Alejandro Agapito Bautista
  • 1,344
  • 1
  • 15
  • 35

1 Answers1

6

You can use Kafka Connect's SMT (Single Message Transforms) feature by adding following code to connect-file-source configuration file.

transforms=createKey
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=UserId <name of user id column>

More information about SMT here

Quang Vien
  • 308
  • 2
  • 10
  • 1
    You can also see an example of this in action here: https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/ – Robin Moffatt Jun 22 '18 at 09:16