1

Is a write to a Kafka topic only successful if the write on each partition's replicas is successful? Or is there a quorum that could be configured?

Suppose you have

  1. Producer
  2. Server1 with Topic1 Partition1(Leader)
  3. Server2 with Topic1 Partition1(Replica)
  4. Server3 with Topic1 Partition1(Replica)

Producer writes to Topic1. Is it a rule the message is stored only if an acknowlegment is received from Leader and both Replicas? Or is it possible to configure a quorum: only leader?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55

2 Answers2

2

If I understand your questions correctly you are looking for the configuration of the Producer API called acks.

acks: The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:

acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1.

acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.

acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.

Type: stringDefault: 1Valid Values: [all, -1, 0, 1]Importance: high

Check documentation for Producer API for more details.

Community
  • 1
  • 1
Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • "The offset given back for each record will always be set to -1" - does the offset given to producer, not to consumer? – J.J. Beam Mar 06 '20 at 20:33
  • 1
    Yes, there is no offset that can be given back to the producer as `acks=0` is the fire-and-forget mode. As the producer is not waiting on a broker response for the write the broker can't give back the new offset. – Michael Heil Mar 06 '20 at 20:42
  • is it possible to configure a quorum? ( e.g. suppose 5 replicas and quorum = 3)? – J.J. Beam Mar 06 '20 at 20:50
  • 1
    This is an interesting question. It can't be done with the `acks` configuration (only the values 0, 1 and all are allowed). I am not aware of how to do 3 out of 5. – Michael Heil Mar 06 '20 at 21:15
  • 1
    why would you want/need that? My point is that I don't think you'll ever need that as a requirement. Usually what you want with having replicas is consistency/durability which is given with the current options. Quorums are susceptible to split-brain situations hence should be avoided unless really necessary. – groo Mar 06 '20 at 22:15
  • you're right, I don't need it, I just wanna answer on interviewer questions... – J.J. Beam Mar 06 '20 at 22:20
1

Aside from acks, there is the question of when a message is visible to consumers - that is independent of acks to producers and is down to broker configuration - it happens when all followers have received the message if they are in sync. if any have fallen behind then it is when min.insync.replicas have received the message. I've written a bit about this at https://chrisg23.blogspot.com/2020/02/kafka-acks-configuration-apology.html?m=1

Chris
  • 1,644
  • 1
  • 11
  • 15