How to debug AvroConsumer in confluent kafka?

Question

I'm trying to read Kafka from python but recieve message is None , No errors in CLI. I'm using port forwarding to destination host via putty, and than test ports over telnet - it work's fine. Moreover, i'm using kafkacat on Debian (WSL) and it work's fine!

kafkacat -C -b localhost:9092 -t topic1 -p 0 -o beginning -s avro -r http://localhost:28081

I'm using PyCharm, my code is below in the text. How do I debug?

from confluent_kafka.avro import AvroConsumer
from confluent_kafka import TopicPartition
from confluent_kafka.avro.serializer import SerializerError

topics = ['topic1', 'topic2']
c = AvroConsumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'smallest',
    'schema.registry.url': 'http://localhost:28081',
    'api.version.request': True
})

c.subscribe(topics)
tp = TopicPartition(topics[0], 0, 0)
c.assign([tp])

while True:
    try:
        msg = c.poll(1)

    except SerializerError as e:
        print("Message deserialization failed for {}: {}".format(msg, e))
        break

    if msg is None:
        print('Message None')
        continue

    if msg.error():
        print("AvroConsumer error: {}".format(msg.error()))
        continue

    print(msg.value())

c.close()

as

score 1 · Answer 1 · answered Aug 13 '20 at 08:54

The first thing I'll do is make sure that messages are coming on your topics using the kafka-avro-console-consumer tool.

Then in your app you can try to increase the log level:

c = AvroConsumer({
    # ... your config here
    'log_level': 7,
    'debug': 'all',
})

You can see the different parameters here: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

But I believe your issue has to to with the way you are assigning partitions. If you use subscribe than partitions are assigned to your consumer automatically by the cluster. You can add a call back when you subscribe to you can see which partitions are assigned to your consumer, but you shouldn't have to do it your self. See https://docs.confluent.io/3.1.1/clients/confluent-kafka-python/index.html#confluent_kafka.Consumer.subscribe

Thanks for debug string! I add comment to strin `#tp = TopicPartition(topics[0], 0, 0) #c.assign([tp])` and now i have debug message, where i see offset 6 msg, but i don't understand why it's not worked. [link_bug_text](https://pastebin.com/8U84K1qZ) . @Arthur, may you help? — shabelski89, Aug 13 '20 at 10:31

How to debug AvroConsumer in confluent kafka?

1 Answers1