How can I initate retries on producer.poll() from the confluent_kafka library?

Question

I'm very new to using kafka, not sure if I understand it yet tbh. However, I'm tasked with making sure it will retry if there is an error of any kind. I've done this is with an API if I recieve a response other than 200 for example, but I'm stumpped with this kafka stuff. Here is my code below if it helps:

def send_kafka_message(payload, secrets):
"""Parses payload into message and sets on kafka producer."""
logger.info("Sending message on Kafka topic...")
producer = Producer(
    {
        "bootstrap.servers": CONFLUENT_SERVER,
        "sasl.mechanism": "PLAIN",
        "security.protocol": "SASL_SSL",
        "sasl.username": CONFLUENT_USER,
        "sasl.password": secrets["confluent_secret"],
        "error_cb": error_cb,
    }
)
producer.produce(
    ORDER_TOPIC,
    value=json.dumps(payload),
    callback=delivery_report,
    partition=abs(hash(payload["_id"])) % 6,
)
producer.poll()

The Producer will automatically retry failed produce requests up to `message.timeout.ms` . — teedak8s, Jun 14 '22 at 19:27
It is just another config option like your security stuff. Search `message.timeout.ms` - https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md — OneCricketeer, Jun 14 '22 at 20:20
By the way, you might want to use `key` parameter rather than `partition`. Also, you should use the murmur2 hash (mentioned in the link) rather than python built-in hash functions if you want to stay consistent with other Kafka clients — OneCricketeer, Jun 14 '22 at 21:20
@OneCricketeer thanks for your suggestions. Might I ask what will be the main advantage of useing "key"? — YourMomsDataEngineer, Jun 16 '22 at 14:38

score 3 · Answer 1 · answered Jun 14 '22 at 20:22

The default retries amount is 2147483647

Description: How many times to retry sending a failing Message. Note: retrying may cause reordering unless enable.idempotence is set to true.

You can find all librdkafka settings, which are used for confluent-kafka-python here.

How can I initate retries on producer.poll() from the confluent_kafka library?

1 Answers1