5

What parameters of Kafka Producer config are needed to be changed so that the producer should: 1) Retry n times 2) After n interval for the same message in case if the broker is down.

I need to handle a situation related to this: https://github.com/rsyslog/rsyslog/issues/1052

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tejas
  • 51
  • 1
  • 1
  • 4

3 Answers3

16

You can set "retries" to n(number of times). But it is not enough you need to look into other configurations also which might get affected because of this or not making it effective.

1) if you have acks = 0 for producer then it will not work. Because acks = 0 → The producer does not wait for any kind of acknowledgment. In this case, no guarantee can be made that the record was received by the broker. retries config does not take effect as there is no way to know if any failure occurred.

2) if you are looking for ordered delivery of event(s) then you need to set max.in.flight.requests.per.connection to 1.

3) retry.backoff.ms - The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

4) request.timeout.ms - The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

5) delivery.timeout.ms - An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgment from the broker (if expected), and the time allowed for retriable send failures. The producer may report failure to send a record earlier than this config if either an unrecoverable error is encountered, the retries have been exhausted, or the record is added to a batch which reached an earlier delivery expiration deadline.

Reference link: https://kafka.apache.org/documentation/#producerconfigs

Skull
  • 1,204
  • 16
  • 28
0

Assuming you're referring to the confluent-kafka-python library, I believe the config you're looking for are:

  1. message.send.max.retries
  2. retry.backoff.ms

See configuration details here

quickinsights
  • 1,067
  • 12
  • 18
-3

I'm not sure if you can really control the no of retries from producer end.

Let me put my understanding here. The producer would trigger the first retry after 'request.timeout.ms' if the response is not received before this timeout elapses and the subsequent retries would be at an interval of 'retry.backoff.ms' till you reach the 'delivery.timeout.ms'

You can find more details here https://kafka.apache.org/documentation/#producerconfigs

Raj
  • 1,467
  • 3
  • 23
  • 51