0

I wonder why Kafka had “acks” (Producer’s config for acknowledgement of delivery) as 1 by default rather than “all” until version 3.0 when the latter offers high durability and hence consistency across all the replicas?

As per my understanding,choosing “all” adds some latency to the acknowledgement at producer’s side but the total end-to-end latency from producer to consumer remains same as is with the other two options for this property.

Durability and reliability should always win over latency, especially in this case where fault tolerance is needed for successful delivery of messages.

Please advise.

This question is based on a configuration property.

user11672850
  • 11
  • 1
  • 4

1 Answers1

0

By default, the acknowledgement setting is set to "1" in Kafka, which means the producer will receive acknowledgement after the leader broker has received the message. This setting is known as "acks=1" or "acks=leader".

The reason behind this default setting is to strike a balance between reliability and performance. When the producer receives acknowledgement from the leader broker, it knows that at least one broker has successfully received the message. This provides a certain level of reliability without sacrificing too much performance.

Setting the acknowledgement to "all" (acks=all) would require acknowledgement from all in-sync replicas (ISRs) before considering the message as successfully sent. This setting offers the highest level of reliability because it ensures that the message has been replicated to all brokers in the ISR. However, it comes at the cost of increased latency and potential performance impact.

Choosing the appropriate acknowledgement setting depends on the specific use case and the desired trade-off between reliability and performance. If data loss is unacceptable and durability is the utmost priority, setting acks to "all" would be appropriate. On the other hand, if performance is critical and a small chance of data loss is acceptable, the default setting of "1" provides a good balance.

Rohit Anil
  • 236
  • 1
  • 11
  • Thank you Rohit. Is there a way to estimate the latency with a given number of ISRs and for a given number of messages - say if 10K messages are pushed by the producer with ISR being 4 then how much time it would take for the producer to receive the acknowledgement from Kafka cluster when acks = 1 vs all? – user11672850 Jun 05 '23 at 21:45
  • I am trying to know how to get the impact caused by one vs the other in terms of latency in milliseconds. Is testing the only way to know this? My understanding is that a very few ISRs shouldn't impact latency by a noticeable margin and so one should always go with option "all" in that case; Also, the consumers consume and process the messages only when all ISRs have received the messages from the producer, so the end-to-end latency of message delivery and consumption remains same with any acks value. – user11672850 Jun 05 '23 at 21:56
  • That's why I want to learn why Kafka is worried about producer getting the acknowledgement from the cluster (broker and/or other replicas) - meaning why is it part of Kafka design? – user11672850 Jun 05 '23 at 21:56
  • Will have to do a test run to see what is the latency but AFAIK, acks should be configured based on the system requirements. For example, a system that requires data consistency like financial systems should go for “all” as the acks value whereas an website impression system can do with 1 as acks value, just an example. – Rohit Anil Jun 06 '23 at 18:20