1

I have upgraded from confluent 4.0.1 to confluent 5.0.1 recently. The bootstrap.server's version is Kafka 1.0. In my HBaseSink Connector, I have configured the new feature ""errors.deadletterqueue.topic.name" as follows:

{ "name": "kafka-hbase4", "config": { "connector.class": "com.xxxx.hbase.sink.HBaseSinkConnector", .... "zookeeper.quorum": "xxxx:2181", "topics": "oplog_demo.hbase_test1", "errors.deadletterqueue.topic.name":"error-topic-1" } }

the program normally works O.K. However, when I send some error messages to the topic of the HBase Sink Connector, like some Chineses words "哈哈哈", the program goes down, throwing a SerializationException, and no messages are consumed by the "error-topic-1".

Does that mean the configuration "errors.deadletterqueue.topic.name" not working? How come could that be and how can I make this configuration work?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
SkyOne
  • 188
  • 3
  • 15
  • Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured – OneCricketeer Nov 26 '18 at 07:10
  • @cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens. – SkyOne Nov 26 '18 at 07:32
  • I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes – OneCricketeer Nov 26 '18 at 18:03
  • 1
    this problem is due to my java classpath set wrong. After I set the right path, problem solved. – SkyOne Nov 27 '18 at 12:23
  • Cool. Feel free to provide the full solution below as an answer rather than a comment – OneCricketeer Nov 27 '18 at 17:33

1 Answers1

0

Make sure that error tolerance is set correctly

"errors.tolerance": "all"

This is a really good resource for DLQ configuration Connecting to Apache Kafka Kafka Connect Deep Dive – Error Handling and Dead Letter Queues

Eric Ekong
  • 107
  • 1
  • 3
  • 8