kafka-python: how to check if broker list available in advance instead of showing error logs

Question

My snippet looks like this, and the logic is if the brokers are available then the producer is not None.

     try:
        self.producer = KafkaProducer(bootstrap_servers=broker_list)
     except kafka.errors.NoBrokersAvailable:
        self.producer = None

But, in this way, there will be a lot of logs printed if the brokers are not available. How can I avoid this?

2020-03-12 13:42:01 INFO <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]>: connecting to device2:9092 [('192.168.0.102', 9092) IPv4]
2020-03-12 13:42:01 INFO Probing node bootstrap-1 broker version
2020-03-12 13:42:01 ERROR Connect attempt to <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]> returned error 111. Disconnecting.
2020-03-12 13:42:01 INFO <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]>: Closing connection. KafkaConnectionError: 111 ECONNREFUSED
2020-03-12 13:42:01 INFO <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]>: connecting to device2:9092 [('192.168.0.102', 9092) IPv4]
2020-03-12 13:42:01 ERROR Connect attempt to <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]> returned error 111. Disconnecting.
2020-03-12 13:42:01 INFO <BrokerConnection node_id=bootstrap-1 host=device2:9092 <connecting> [IPv4 ('192.168.0.102', 9092)]>: Closing connection. KafkaConnectionError: 111 ECONNREFUSED
2020-03-12 13:42:01 INFO <BrokerConnection node_id=bootstrap-0 host=device1:9092 <connecting> [IPv4 ('192.168.0.101', 9092)]>: connecting to device1:9092 [('192.168.0.101', 9092) IPv4]

score 0 · Accepted Answer · answered Mar 12 '20 at 08:12

Not really, brokers are considered unavailable if they cannot be reached and/or they do not respond to ApiVersions requests. So to check if servers are alive you'd need to establish a connection, send a request, and check if response returned is valid.

However, trivial checking whether all of the _bootstrap_servers_ are connectable might be sufficient - this way you could quickly eliminate e.g. firewall issues.

On the other hand, you might investigate logging levels for these classes.

score 0 · Answer 2 · answered Mar 12 '20 at 08:20

0

You can use KazooClient to establish a connection with Zookeeper and then list the available brokers in the cluster.

answered Mar 12 '20 at 08:20

Mohamed Zouari

395
1
5
14

Don't use Zookeeper anymore. Use the AdminClient – OneCricketeer Mar 12 '20 at 13:52

kafka-python: how to check if broker list available in advance instead of showing error logs

2 Answers2