Log offset, partition and topic from Kafka message

Question

I want to log or print offset, partition, and topic from Kafka's message. I can print the message value but I want to see which offset and partition from Kafka using python so that I can debug my code

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['localhost:9092'])
for message in consumer:
    print(f"Message is {message.value()}")

score 2 · Accepted Answer · answered Oct 20 '21 at 13:21

2

That data should be available using dot notation

The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value:

https://github.com/dpkp/kafka-python

answered Oct 20 '21 at 13:21

OneCricketeer

179,855
19
132
245

I added an example code where I can print messages so can you help me with how to print offset, topic, and partition for the message, please. – Rahul Oct 20 '21 at 14:40
I assume you have `print(msg.value)`? What issues are you having with `print(msg.topic, msg.partition, msg.offset)`? – OneCricketeer Oct 20 '21 at 14:41
When i print(msg.value) i get output as – Rahul Oct 20 '21 at 14:46
Well, [documentation shows `msg.value` is a dict when deserializing](https://github.com/dpkp/kafka-python#kafkaconsumer), not a function... If it is a function, then use `msg.value()` like you've shown in your question. You need to call the function... You could also `print(dir(msg))` to see all available attributes of the object – OneCricketeer Oct 20 '21 at 14:49
for msg.value() is working fine but I want to print offset, partition and for print(dir(msg)) I get output as ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'error', 'headers', 'key', 'latency', 'offset', 'partition', 'set_headers', 'set_key', 'set_value', 'timestamp', 'topic', 'value'] – Rahul Oct 20 '21 at 14:52
When i print(msg.offset) or print(msg.partition) i get output as – Rahul Oct 20 '21 at 14:53
That output is for `.topic`, not offset or partition... Again, as the output shows, those are methods that you need to call – OneCricketeer Oct 20 '21 at 16:02
1

logger.debug(f"Consumer data: offset-{message.offset()} partition-{message.partition()} topic-{message.topic()}") This worked for me thank you – Rahul Oct 20 '21 at 17:17

score 0 · Answer 2 · answered Oct 20 '21 at 15:38

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['localhost:9092'])
for message in consumer:
    logger.debug(f"Consumer data: offset-{message.offset()} partition-{message.partition()} topic-{message.topic()}")
    print(f"Message is {message.value()}")

Log offset, partition and topic from Kafka message

2 Answers2