3

So i've been reading about kafka's exactly once semantics, and I'm a bit confused about how it works.

I understand how the producer avoids sending duplicate messages (in case the ack from the broker fails), but what I don't understand is how exactly-once works in the scenario where the consumer processes the message but then crashes before committing the offset. Won't kafka retry in that scenario?

Atul Bhatia
  • 1,633
  • 5
  • 25
  • 50

2 Answers2

6

here's what i think you mean:

  1. consumer X sees record Y, and "acts" on it, yet does not commit its offset
  2. consumer X crashes (still without committing its offsets)
  3. consumer X boots back up, is re-assigned the same partition (not guaranteed) and eventually sees record Y again

this is totally possible. however, for kafka exactly once to "work" all of your side effects (state, output) must also go into the same kafka cluster. so here's whats going to happen:

  1. consumer X starts a transaction
  2. consumer X sees record Y, emits some output record Z (as part of the transaction started in 1)
  3. consumer X crashes. shortly after the broker acting as the transaction coordinator "rolls back" (im simplifying) the transaction started in 1, meaning no other kafka consumer will ever see record Z
  4. consumer X boots back up, is assigned the same partition(s) as before, starts a new transaction
  5. consumer X sees record Y again, emits record Z2 (as part of the transaction started in 4)
  6. some time later consumer X commits its offsets (as part of the transaction from 4) and then commits that transaction
  7. record Z2 becomes visible to downstream consumers.

if you have side-effects outside of the same kafka cluster (say instead of record Z you insert a row into mysql) there's no general way to make kafka exactly-once work for you. you'd need to rely on oldschool dedup and idempotance.

radai
  • 23,949
  • 10
  • 71
  • 115
  • ah got it -- so i can set up an external database (which is where record Z is written to) as part of the kafka cluster? lets assume the db is mysql – Atul Bhatia Aug 19 '19 at 05:20
  • Kafka doesn't offer a specific solution to sync message commits with a relational database, therefore it must be handled manually by developer. I suggest reading a paragraph from the book "Kafka: The Definitive Guide". In chapter 4, paragraph "Consuming Records with Specific Offsets", is explained this exact problem. – Gabriele Coletta Aug 19 '19 at 10:20
  • @radai How does consumer start transactions? Transaction APIs are for producer to use. In case of read-process, when consumer restart how to make sure it doesn't read the same message again? Are you referring that It need to be handled manually, saving offset in db and when consumer come back and check the last read offset and then can read from there onwards? – Girish Jan 03 '22 at 10:27
  • check out the isolation.level configuration on consumers - https://kafka.apache.org/documentation/#consumerconfigs_isolation.level consumers do not start/commit/rollback any transactions. but they can definitely avoid giving you back any records until the relevant TX has been committed. also consumer offset commits CAN be done under a transaction – radai Jan 06 '22 at 23:55
  • 1
    Can someone explain where is this transaction co-ordinator living and how will this transaction co-ordinator removes the record Z explained above? – Sachin Sep 08 '22 at 21:12
0

Radal explained it well in its answer, regarding exactly once in a isolated Kafka cluster.

When dealing with an external database ( transactional at least) , one easy way to achieve exactly once is to UPDATE one row ( in a sgbd transaction), with your business value AND the Partition / offsets where it comes from. That way , if your consumer crash before committing to Kafka, you'll be able to get back the last Kafka offset it has processed ( by using consumer.seek())

It can though be a quite data overhead in your sgbd ( keeping offset/partition for all your rows), but you might be able to optimize a bit.

Yannick

Yannick
  • 1,240
  • 2
  • 13
  • 25