9

Read this article about message ordering in topic partition: https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2

Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.

According it there are two types of producer configs possible to achieve ordering guarantee:

max.in.flight.requests.per.connection=1 // can impact producer throughput

or alternative

enable.idempotence=true
max.in.flight.requests.per.connection //to be less than or equal to 5
max.retries // to be greater than 0
acks=all

Can anybody explain how second configuration achieves order guarantee? Also in the second config exactly-once semantics enabled.

bausov
  • 387
  • 3
  • 12

2 Answers2

8

idempotence:(Exactly-once in order semantics per partition)

Idempotent delivery enables the producer to write a message to Kafka exactly once to a particular partition of a topic during the lifetime of a single producer without data loss and order per partition.

Idempotent is one of the key features to achieve Exactly-once Semantics in Kafka. To set “enable.idempotence=true” eventually get exactly-once semantics per partition, meaning no duplicates, no data loss for a particular partition. If an error occurred even producer send messages multiple times will get written to Kafka once.

Kafka producer concept of PID and Sequence Number to achieve idempotent as explained below:

PID and Sequence Number

Idempotent producers use product id(PID) and sequence number while producing messages. The producer keeps incrementing the sequence number on each message published which map with unique PID. The broker always compares the current sequence number with the previous one and it rejects if the new one is not +1 greater than the previous one which avoids duplication and the same time if more than greater show lost in messages.

enter image description here

In a failure scenario it will still maintain sequence number and avoid duplication as shown below:

enter image description here

Note: When the producer restarts, new PID gets assigned. So the idempotency is promised only for a single producer session

If you are using enable.idempotence=true you can keep max.in.flight.requests.per.connection up to 5 and you can achieve order guarantee which brings better parallelism and improve performance.

Idempotence feature introduced in Kafka 0.11+ before we can achieve some level level of guaranteed using max.in.flight.requests.per.connection with retries and Acks setting:

max.in.flight.requests.per.connection to 1
max.retries bigger number
acks=all

max.in.flight.requests.per.connection=1: to make sure that while messages are retrying, additional messages will not be sent.

This gives guarantee at-least-once and comes with cost on performance and throughput and that's encourage introduced enable.idempotence feature to improve the performance and at the same time guarantee ordering.

exactly_once: To achieve exactly_once along with idempotence we need to set transaction as read_committed and will not allow to overwrite following parameters:

  • isolation.level:read_committed( Consumers will always read committed data only)

  • enable.idempotence=true (Producer will always haveidempotency enabled)

  • MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5 (Producer will always have one in-flight request per connection)

Nitin
  • 3,533
  • 2
  • 26
  • 36
  • I don't think this answers the question. Of course you showed the easy case, when `seq=3` fails and you retry. The much more interesting part is when you have 5 requests, the first 2 are committed, the 3-rd fails and thus retries, and the 4 and 5-th are... committed? All of the requests for the same partition. what will happen then, when the 3-rd is retried? more interesting, what if there are consumers? what and how will they read these messages, if they will at all. This answer is kind of superficial. – Eugene Jan 21 '22 at 12:13
  • of course in my case above, the broker should not accept the 4-th and 5-th requests at all, since the 3-rd has not arrived. But a) this means every single messages for this partition is delayed until the retry succeeds? b) what if it does not succeeded at all, ever? are the PID and seq somehow reset? – Eugene Jan 21 '22 at 12:15
  • and my last point from the KIP of the change itself: "When idempotence is enabled, we enforce that acks=all, retries > 1, and max.inflight.requests.per.connection=1.". So when `idempotence` is enabled, `max.inflight.requests.per.connection=1` is forced. There is simply no other way to handle this. – Eugene Jan 21 '22 at 12:23
  • @Eugene, looks like this was changed https://issues.apache.org/jira/browse/KAFKA-5494 – ievgen.garkusha Feb 22 '22 at 13:47
  • @ievgen.garkusha indeed, thank you! this is something I do not like about kafka, absolutely terrible documentation, especially when you start hitting points like this. again, much appreciate the link! – Eugene Feb 22 '22 at 19:33
3

enable.idempotence is a newer setting that was introduced as part of kip-98 (implemented in kafka 0.11+). before it users would have to set max.inflight to 1.

the way it works (abbreviated) is that producers now put sequence numbers on ourgoing produce batches, and brokers keep track of these sequence numbers per producer connected to them. if a broker receives a batch out of order (say batch 3 after 1) it rejects it and expects to see batch 2 (which the producer will retransmit). for complete details you should read kip-98

radai
  • 23,949
  • 10
  • 71
  • 115
  • 1
    thanks for the reference. So I found, that in case of order violation broker throws fatal exception. And there is no mechanism to resend previous batch and then continue keeping right order. The Producer will raise an OutOfOrderSequenceException if the broker detects data loss. In other words, if it receives a sequence number which is greater than the sequence it expected. This exception will be returned in the Future and passed to the Callback, if any. This is a fatal exception, and future invocations of Producer methods like send, beginTransaction etc. will throw IllegalStateException – bausov Nov 07 '19 at 19:46
  • thank you for the KIP reference! There are waaay too many wrong assumptions and answers while the KIP precisely says : "When idempotence is enabled, we enforce that acks=all, retries > 1, and max.inflight.requests.per.connection=1. Without these values for these configurations, we cannot guarantee idempotence. If these settings are not explicitly overidden by the application, the producer will set acks=all, retries=Integer.MAX_VALUE, and max.inflight.requests.per.connection=1 when idempotence is enabled.". – Eugene Jan 21 '22 at 12:26