0

I used kafka-python to process messages in a kafka cluster:

consumer = KafkaConsumer('session', auto_offset_reset='earliest']

while True:

   dict = consumer.poll(500)

   for d in dict:

     print d.topic, d.partition, d.value

That will give err "AttributeError: 'TopicPartition' object has no attribute 'value'".

"dict" is like this (from 'print dict')

{TopicPartition(topic=u'session', partition=0): [ConsumerRecord(topic=u'session', partition=0, offset=56, timestamp=None, timestamp_type=None, key=None, value='0000000000000000', headers=[], checksum=2855809697, serialized_key_size=-1, serialized_value_size=16, serialized_header_size=-1)]}

There can be many partitions and hundreds of ConsumerRecord's under each partition. What is the right way(s) to access those ConsumerRecord's from consumer.poll()? Thanks in advance.

Sreekiran A R
  • 3,123
  • 2
  • 20
  • 41
richardj
  • 11
  • 1
  • 3

1 Answers1

1

You have a mistake at dict usage; By default, "for d in dict:" means "for d in dict.keys():", so you can only fetch the keys of this dict. try this:

dict = consumer.poll(500)
for key, value in dict.items():
    print(key)
    print()
    for record in value[:10]:
        print(record)
        print()

This could fix your error.

Zhang Shu
  • 11
  • 2