Python confluent kafka raise exception on broker connection disconnect

Question

I am using python 3.7 and confluent-kafka.

Following is the pseudo code which I am using to poll the kafka server and read the message.

        while True:
            MSG = CONSUMER.poll(0.1)
            if MSG is None:
                CONSUMER.commit()
                print('No msg')
                continue
            if MSG.error():
                print("Consumer error: {}".format(MSG.error()))
                CONSUMER.commit()
                continue
            try:
                rawMsg = format(MSG.value().decode('utf-8'))
                testmsg = json.loads(rawMsg)
            except:
                print('invalid json format msg')
                CONSUMER.commit()

If the kafka server is down or disconnected for some reason, I want an exception to be thrown.

Currently, if the above happens, the while loop keeps running without any error and prints No msg.

How do I get an exception or check if the kafka server can be connected each time in the loop (If there is some check to be made it should be lightweight).

https://stackoverflow.com/questions/61226910/how-to-programmatically-check-if-kafka-broker-is-up-and-running-in-python — Kimi, Jul 29 '21 at 17:50

score 1 · Answer 1 · answered Mar 31 '22 at 13:01

When creating a Consumer you can specify in the deserializer a callback for error.

Here is an example using same mechanism in a producer :

import confluent_kafka
def error_callback(err):
    print("callback hit!")
    raise(err)
p = confluent_kafka.Producer({
    "bootstrap.servers": "localhost:9092",
    "message.max.bytes": 10_000_000,
    "error_cb": error_callback,
    "debug": "msg",
})
p.produce("test-topic", "a" * int(2e6))
p.flush()

From github issues. It may help but it doesn't solve the case.

Python confluent kafka raise exception on broker connection disconnect

1 Answers1