I'm trying to configure exactly once semantics in Kafka (Apache Beam). Here are the changes what I'm going to introduce:
Producer:
enable.idenpotence
= truetransactional.id
= uniqueTransactionalId
Consumer:
set
enable.auto.commit
= false// added the following to consumer builder:
.commitOffsetsInFinalize()
.withReadCommitted()
Added the following to KafkaIO#write
builder:
.withEOS(numShards, sinkGroupId)
Does anyone know what else should be changed to achieve exactly once semantics in Apache Beam KafkaIO?
Does the configuration above looks fine or I misunderstood smth?
Do I need to specify the transactional.id
property if I do not use transaction API (because I do not have explicit producer in apache beam)?