0

How should the transaction requires abort case be handled for the Transactional Producer API?

According to the documentation, if while trying to commit a transaction, an error occurs where a transaction should be aborted (possibly due to rebalancing), the steps to be taken should be:

  • producer --> abort transaction
  • producer --> begin transaction
  • rewind consumer offsets

Aborting and beginning a new transaction are straight forward. However, how and how far should the consumer offset be rewound? What would this look like for the Python client? In the case of a single message being consumed at a time, should only that message simply be reprocessed?

As a reference, the code example that I'm referring to is:

while True:
   try:
       producer.commit_transaction(10.0)
       break
   except KafkaException as e:
       if e.args[0].retriable():
          # retriable error, try again
          continue

       # **************Relevant to the question**************
       elif e.args[0].txn_requires_abort():
          # abort current transaction, begin a new transaction,
          # and rewind the consumer to start over.
          producer.abort_transaction()
          producer.begin_transaction()
          rewind_consumer_offsets...()
       # **************

       else:
           # treat all other errors as fatal
           raise
Moritz
  • 2,987
  • 3
  • 21
  • 34
  • Producers shouldn't be responsible for modifying consumers. They would simply commit offsets. I see a function in the docs "send offsets to transaction" – OneCricketeer Feb 06 '22 at 14:05
  • Do you mean, only commit the producer's offset? AFAIK, they are responsible when using the Transactional API. According to [example code](https://github.com/apache/kafka/blob/3.0/examples/src/main/java/kafka/examples/ExactlyOnceMessageProcessor.java) all TopicPartitions that a consumer subscribes to should be reset to their committed index. – Moritz Feb 07 '22 at 15:42
  • Not sure I understand the question. That code just calls `consumer.seek`, and doesn't really matter if there are transactions, or not – OneCricketeer Feb 07 '22 at 15:50
  • 1
    But if you are trying to copy that example into Python, I was referring to the line `producer.sendOffsetsToTransaction` which maps to similar `send_offsets_to_transaction` function – OneCricketeer Feb 07 '22 at 15:58

0 Answers0