0

I am using AWS MSK Kafka managed streaming service and all of a sudden I am not receiving any messages through my consumer. I am using kafka-python and the weird thing is that when I open up my cli and create a new consumer with the exact same code, I receive the messages just fine. If I restart the kafka broker, only then will I get messages. Any idea what the issue could be?

>>> consumer = KafkaConsumer(
...                 bootstrap_servers=[...],
...                 auto_offset_reset='earliest',
...                 enable_auto_commit=True,
...                 group_id='my-client-id',
...                 api_version=(2, 2, 1))
>>> consumer.subscribe(['hello_world', 'foo_bar'])
>>> for message in consumer:
...     print(message)
joethemow
  • 1,641
  • 4
  • 24
  • 39
  • Does this answer your question? [Python Kafka consumer doesn't receive the message from beginning?](https://stackoverflow.com/questions/64054801/python-kafka-consumer-doesnt-receive-the-message-from-beginning) – Michael Heil Sep 29 '20 at 05:19
  • Unfortunately not, I don't want to read from the beginning all the time. Instead just pick up right after the latest message. But regardless the bigger issue is that I am not receiving and data – joethemow Sep 29 '20 at 05:42
  • okay, can you be more specific what you mean with "exact same code". I am assuming when you say "cli" you mean command line interface so I don't understand how this can be exactly the same as it is not Python code. Can you show the command line command? – Michael Heil Sep 29 '20 at 06:31
  • I figured out the issue, I actually call a function inside the for loop, but it was not returning or got hung so the thread got blocked. I have the KafkaConsumer already running on its own process. Any tips to possibly have a timeout on the function? Or just run it on its own thread? – joethemow Sep 29 '20 at 07:31
  • This is really difficult to say without seeing the code. Also, I have to admit that I am not very familiar with Python API... – Michael Heil Sep 29 '20 at 07:52

0 Answers0