Ensure exactly once delivery in Kafka Connect

Question

do You know is it possible, and if yes what is the best way to ensure exactly one delivery to hdfs using kafka connect with kafka?

I know that Kafka connect attempt to find offsets for its consumer group in the "'__consumer_offsets" but I need additional check as duplicates is not acceptable

score 0 · Answer 1 · answered May 14 '19 at 14:12

0

HDFS Connect already claims to support exactly once by using a write ahead log in HDFS. When connect is restarted, it actually checks that log, unless the logic recently changed, not the offsets topic

answered May 14 '19 at 14:12

OneCricketeer

179,855
19
132
245

And by log You mean latest HDFS file? – Rafał Wójcik May 15 '19 at 08:03
There are multiple files. There is a log file that connect writes on HDFS, as well as temporary and final files where your data would be written – OneCricketeer May 15 '19 at 18:56

score 0 · Answer 2 · answered May 14 '19 at 16:59

When connector writes files to HDFS it first writes to temp file,WAL for replay and then rename temp to final file . The naming of this final file has the offsets that is present in that file. So when connect starts up it looks on HDFS and finds the latest committed offset which should guarantee once only delivery. If an offset is not found in hdfs then it lets consumers offset reset policies .Take a look at https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/DataWriter.java and https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/TopicPartitionWriter.java to understand more

Ensure exactly once delivery in Kafka Connect

2 Answers2