I'm trying to send messages using kafka-python to my AWS MSK Serverless cluster. I've created MSK Serverless cluster by following this guide: https://docs.aws.amazon.com/msk/latest/developerguide/msk-serverless-produce-consume.html
My cluster is in active state and I have an EC2 instance in the same VPC and with the same subnet as my cluster. I've created a topic an successfully sent and red messages to it and from it by using the command line (Step 5 in the tutorial).
Now I need to do the same using python. So I tried to do the following:
from kafka import KafkaProducer
cluster_endpoint = 'boot-name.c1.kafka-serverless.reagion-name.amazonaws.com:9098'
producer = KafkaProducer(security_protocol="SSL",bootstrap_servers=[cluster_endpoint])
producer.send('my-topic-name', "test message")
But at this point I receive
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/.local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 576, in send
self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0)
File "/home/ec2-user/.local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 703, in _wait_on_metadata
"Failed to update metadata after %.1f secs." % (max_wait,))
kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.
I've noticed that usually the port is 9091
or 9092
but when I go to "View client information" for my cluster my endpoint is as I wrote in my code with 9098
port number.
I tied to follow different advises here but nothing had helped. This is what I've tried:
- KafkaTimeoutError: Failed to update metadata after 60.0 secs
- KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 300.0 secs
- https://community.bitnami.com/t/kafka-producer-timeoutexception-failed-to-update-metadata-after-60000-ms/82159/6
What am I missing?