1

I am trying to write a custom source connector for kafka connect and it is requiring for me to write a simple plain kafka producer inside it (that has no relevance with any of the connector properties).The requirement is to send messages about bad records that I encounter and send it to another topic using a plain producer.

So in order to start the producer I need the bootstrapserver config that was originally used to start the connect worker.

Is it possible to dynamically read the connect-worker config inside a connector?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jenison Gracious
  • 486
  • 4
  • 13
  • A Source connector already is a producer and bad records can be captured using the `errors.tolerance` configurations. https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect – OneCricketeer Oct 17 '19 at 17:04
  • @cricket_007 lets say there is another requirement for me to monitor the number of records processed (to send updates at frequent intervals) that has no connection with the connector data. For this scenario it is better to have a simple producer (for which I need the bootstrapserver properties of the connect cluster) – Jenison Gracious Oct 22 '19 at 09:15
  • 1
    I'm not sure bootstrap servers or other properties can be accessed (outside of reading the connect property file off disk or duplicating the information to the task configs). – OneCricketeer Oct 22 '19 at 14:22

0 Answers0