0

I'm trying to build a KafkaConnect source connector and was wondering if it is possible to inject/use a custom Kafka Producer in the process. As per the documentation, I couldn't get much. Any ideas would help!

guru
  • 409
  • 4
  • 21

1 Answers1

1

The poll method of a SourceTask is the producer. Return a List<SourceRecord>, and they get serialized into the configured topic via the Converter definitions.

If you really want an actual instance of a Producer object, then you shouldn't be using the Connect API

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Yeah, that's what I figured as well. Thanks! – guru Feb 11 '20 at 18:16
  • Then again, there really is nothing stopping you from creating a `new KafkaProducer` in that method, but you would need to configure it separately from the main Connect configurations – OneCricketeer Feb 11 '20 at 18:37
  • Also, another question I have is, where can I provide the bootstrap servers and schema registry configuration to the kafka connector? Meaning, can I have the class that extends AbstractConfig also take the kafka bootstrap servers' details and produce to that cluster? – guru Feb 11 '20 at 18:46
  • If I do configure a separate KafkaProducer in the poll method, what happens to the records that we return from the poll method? Will they not be produced again to the same topic? – guru Feb 11 '20 at 18:53
  • 1
    To clarify: You would not make the producer send to the same topic as Connect. Connect Bootstrap servers are in the `connect-distributed.properties` file. The Schema Regsistry is defined as `(key|value).converter.schema.registry.url`... If you've downloaded Confluent Platform, then you can find examples in the etc/schema-registry directory – OneCricketeer Feb 11 '20 at 19:18
  • Found it, thanks. It's a little confusing as to which configuration is for what purpose. Is the `connect-distributed.properties` file separate than the properties file we provide as part of our source connector? – guru Feb 11 '20 at 22:22