How does kafka ack batch AsyncProducer

Question

How does kafka sends ack when using batch async Producer? Is the ack is per message/per batch/per sub-batch(i.e. batch per partition)? Is it recommended to use the ack in async batch prdocuer? or better just to use the callback mechanism?

Answer 1

How does kafka sends ack when using batch async Producer

two choice by updating se Future, or by doing the callback code.

• java.util.concurrent.Future send​(ProducerRecord<K,V> record)
• java.util.concurrent.Future send​(ProducerRecord<K,V> record, Callback callback)

Is the ack is per message/per batch/per sub-batch(i.e. batch per partition)

Good question. I think the ack is for the entire request so all the messages of the request (but not sure, and I didn't find the info)

Is it recommended to use the ack in async batch prdocuer? or better just to use the callback mechanism?

It is two different notion, send are alway async with kafka if you do not do a future.get() If you want to send many record by batch you will have to send without blocking. (without doing future.get() at each time)

Answer 2

"How does kafka sends ack when using batch async Producer? Is the ack is per message/per batch/per sub-batch(i.e. batch per partition)?"

The ack is sent per batch (which also means: per partition). If one or more individual messages within a batch fails, the entire batch is considered a failure. Depending on your retries configuration this batch will then be re-sent.

"Is it recommended to use the ack in async batch prdocuer? or better just to use the callback mechanism?"

I do not think this is an either-or question. You can have acks>0 and at the same time use a callback. The acks setting in general allows you to improve upon the stability of your Producer, independently of any callbacks.

You should use callback if you require to understand the full details why the sent was a failure. I have provided some more details on Producer Callback Exceptions on another post of mine.

In addition, you could use the callback of your asynchronous producer to handle manual commits while avoid committing lower offsets due to retires as described in Kafka Consumer offset commit check to avoid committing smaller offsets