0

This code doesn't display any messages successfully

What's wrong?

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'topictest',
     bootstrap_servers=['server:9092'],
     auto_offset_reset='None',
     enable_auto_commit=True,
     group_id='test-consumer-group',
     value_deserializer=lambda x: loads(x.decode('utf-8')))
for message in consumer:
    message = message.value
    print('{} added to'.format(message))
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
VLBigo
  • 9
  • 4

1 Answers1

0

Try this

from kafka import KafkaConsumer
from json import loads
consumer = KafkaConsumer(
    'topictest',
     bootstrap_servers=['10.17.48.245:9092'],
     auto_offset_reset='latest',
     enable_auto_commit=True,
     group_id='test-consumer-group',
     value_deserializer=lambda x: loads(x.decode('ISO-8859-1')))

for message in consumer:
     print(message.value)

You can try without changing the encoding, as well

Ritwik Bandyopadhyay
  • 125
  • 1
  • 9
  • Why change the encoding format? – OneCricketeer Oct 17 '22 at 10:50
  • error :( JSONDecodeError Traceback (most recent call last) Cell In [35], line 11 JSONDecodeError: Expecting value: line 1 column 1 (char 0) WARNING:kafka.coordinator:Heartbeat failed for group test-consumer-group because it is rebalancing – VLBigo Oct 17 '22 at 10:54
  • The issue is with the auto_offset_reset or decoding. Sometimes, the IDE or environment being used creates issues. This is assuming that the problem is confined to reading Kafka data in Python. – Ritwik Bandyopadhyay Oct 17 '22 at 10:55
  • @Bogdan Try changing the group_id to test-consumer-group-2 – Ritwik Bandyopadhyay Oct 17 '22 at 10:56
  • 1
    @Ritwik Bandyopadhyay changed, now there is no error, but the script is not executed for a very long time and does not display anything – VLBigo Oct 17 '22 at 11:01
  • @Ritwik Bandyopadhyay WARNING:kafka.coordinator:Heartbeat failed for group test-consumer-group-2 because it is rebalancing WARNING:kafka.coordinator:Heartbeat failed for group test-consumer-group because it is rebalancing – VLBigo Oct 17 '22 at 11:06
  • 1
    very long execution, but there are five messages in one word – VLBigo Oct 17 '22 at 12:53
  • @Ritwik Bandyopadhyay – VLBigo Oct 25 '22 at 06:49
  • Yeah. That can happen if something keeps going into the buffer. You can maybe deduplicate it separately – Ritwik Bandyopadhyay Oct 26 '22 at 05:37