2

I'm trying to use Kafka-Python for a very basic way of streaming some data by setting up a KafkaProducer and a KafkaConsumer in two jupyter notebooks, similar to this article: https://dorianbg.wordpress.com/2017/11/11/ingesting-realtime-tweets-using-apache-kafka-tweepy-and-python/

Setting up my producer:

import time
!pip install kafka-python
from kafka import KafkaConsumer, KafkaProducer

producer = KafkaProducer(bootstrap_servers=['localhost:9092'], api_version=(0, 10, 1), max_block_ms = 300000, api_version_auto_timeout_ms=300000)
topic_name = 'test'


def send_data(interval):

    while True: 
        producer.send(topic_name, str(1))
        time.sleep(interval)

Setting up my consumer:

!pip install kafka-python
from kafka import KafkaConsumer, KafkaProducer

consumer = KafkaConsumer(bootstrap_servers = "localhost:9092",
                        group_id = "random",
                        api_version=(0, 10, 1), consumer_timeout_ms=1000)
consumer.subscribe('test')

for message in consumer:
    print(message)

When I run these in separate notebooks, I get the following message in the producer-notebook:

KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 300.0 secs.

I'm well aware that this is a question that has been asked previously (KafkaTimeoutError('Failed to update metadata after 60.0 secs.'), https://github.com/dpkp/kafka-python/issues/721, etc.) but these situations differ from mine, or the answers provided do not resolve my issues.

Thanks!

Erik Hallin
  • 245
  • 2
  • 6
  • 17

0 Answers0