1

I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.

My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)

So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
SkyOne
  • 188
  • 3
  • 15
  • What do you mean by _Connect's `bootstrap.servers`_ ? Do you want the host of Kafka Connect or Kafka brokers? – Giorgos Myrianthous Nov 20 '18 at 12:57
  • @GiorgosMyrianthous the bootstrap.servers for me to build a Kafka producer... I think that should be the Kafka brokers – SkyOne Nov 20 '18 at 13:08
  • I don't think you can make a call to [REST Proxy API](https://docs.confluent.io/current/kafka-rest/docs/api.html#crest-long-api-reference) in order to get `bootstrap.servers`. [You can only get Brokers' IDs](https://docs.confluent.io/current/kafka-rest/docs/api.html#brokers). I would suggest to create a `.properties` file that contains your `bootstrap.servers` and load that file on producer's startup. – Giorgos Myrianthous Nov 20 '18 at 13:10
  • I don't see anything wrong with reading the file from disk, assuming all Connect workers are consistently installed... In fact, that's how the new password hiding feature works – OneCricketeer Nov 20 '18 at 15:34
  • @cricket_007 What if the file name is various against different environments? i.e. etc/dev/dev-connect-distributed.properties and etc/prd/prd-connect-distributed.properties – SkyOne Nov 20 '18 at 22:50
  • Well, hopefully, you wouldn't be running dev and prod things on the same machine (or more than one Kafka Connect server on the same machine) :) – OneCricketeer Nov 20 '18 at 23:07

1 Answers1

2

Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.

You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.

For more details, see the KIP that added this feature.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Worth pointing out - Connect can be upgraded independently of the brokers – OneCricketeer Nov 20 '18 at 15:32
  • @cricket_007 Our company now use Confluent4.0.1 with Kafka 1.0.1, can we just upgrade Confluent4 to Confluent5 while still using the Kafka 1.0.1 cluster? – SkyOne Nov 21 '18 at 01:58
  • @Sky You should be able to yes. Brokers since Confluent 3.0 support newer Confluent versions for Connect workers – OneCricketeer Nov 21 '18 at 04:38