I'm trying to use Kafka-Python for a very basic way of streaming some data by setting up a KafkaProducer and a KafkaConsumer in two jupyter notebooks, similar to this article: https://dorianbg.wordpress.com/2017/11/11/ingesting-realtime-tweets-using-apache-kafka-tweepy-and-python/
Setting up my producer:
import time
!pip install kafka-python
from kafka import KafkaConsumer, KafkaProducer
producer = KafkaProducer(bootstrap_servers=['localhost:9092'], api_version=(0, 10, 1), max_block_ms = 300000, api_version_auto_timeout_ms=300000)
topic_name = 'test'
def send_data(interval):
while True:
producer.send(topic_name, str(1))
time.sleep(interval)
Setting up my consumer:
!pip install kafka-python
from kafka import KafkaConsumer, KafkaProducer
consumer = KafkaConsumer(bootstrap_servers = "localhost:9092",
group_id = "random",
api_version=(0, 10, 1), consumer_timeout_ms=1000)
consumer.subscribe('test')
for message in consumer:
print(message)
When I run these in separate notebooks, I get the following message in the producer-notebook:
KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 300.0 secs.
I'm well aware that this is a question that has been asked previously (KafkaTimeoutError('Failed to update metadata after 60.0 secs.'), https://github.com/dpkp/kafka-python/issues/721, etc.) but these situations differ from mine, or the answers provided do not resolve my issues.
Thanks!