1

After creating a Connector and a SourceTask objects in Kafka with Java I cannot figure out what can be done with SourceRecords returned form poll() methd of the SourceTask object.

How can I push the record in the topic which is a parameter of the SourceRecocd?

I mean we have the creation of SourceRecords in poll() method like this:

    records.add(new SourceRecord(sourcePartition, sourceOffset, config.topicName, Schema.STRING_SCHEMA, message));

So now how to produce a message in the topic which is the config.topicName above?

Thanks in advance.

Erwin Bolwidt
  • 30,799
  • 15
  • 56
  • 79
Novemberland
  • 530
  • 3
  • 8
  • 25

1 Answers1

1

Field topic from SourceRecord instance point to the topic where message will be published. In your case it should publish message to config.topicName.

Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30
  • You mean that the creation of the object SourceRecord automatically publishes the message to the topic ? – Novemberland Mar 06 '19 at 22:29
  • When you create `SourceRecord` you chose destination topic. Additionally if you need to modify it, you can do it later via `Transformations` – Bartosz Wardziński Mar 06 '19 at 22:31
  • That means if you want first to transform the records and then publish them cannot happen? In this case the creation of SourceRecord object automatically publishes the message to the topic which are parameters and then you decide what to do with the transformed records? – Novemberland Mar 06 '19 at 22:38
  • No, Depending which type of connector the flow is different. For SourceConnector: 1. Records are polled from external system, 2. Records are Transformed, 3. Records are converted to Array of bytes, for SinkConnector flow is reversed. You can read more about concept and architecture: https://docs.confluent.io/current/connect/concepts.html – Bartosz Wardziński Mar 06 '19 at 22:47
  • @Novemberland *automatically publishes the message to the topic* -- Yes. When you `return records` in your code, the internal Connect framework code will eventually create a Producer and send to the topic that is passed to the SourceRecord parameter – OneCricketeer Mar 08 '19 at 16:56