1

I'm trying to send messages using kafka-python to my AWS MSK Serverless cluster. I've created MSK Serverless cluster by following this guide: https://docs.aws.amazon.com/msk/latest/developerguide/msk-serverless-produce-consume.html

My cluster is in active state and I have an EC2 instance in the same VPC and with the same subnet as my cluster. I've created a topic an successfully sent and red messages to it and from it by using the command line (Step 5 in the tutorial).

Now I need to do the same using python. So I tried to do the following:

from kafka import KafkaProducer
cluster_endpoint = 'boot-name.c1.kafka-serverless.reagion-name.amazonaws.com:9098'
producer = KafkaProducer(security_protocol="SSL",bootstrap_servers=[cluster_endpoint])
producer.send('my-topic-name', "test message")

But at this point I receive

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/.local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 576, in send
    self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 703, in _wait_on_metadata
    "Failed to update metadata after %.1f secs." % (max_wait,))
kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.

I've noticed that usually the port is 9091 or 9092 but when I go to "View client information" for my cluster my endpoint is as I wrote in my code with 9098 port number.

I tied to follow different advises here but nothing had helped. This is what I've tried:

What am I missing?

Nina
  • 89
  • 1
  • 7
  • What other properties are in your config file mentioned in the MSK link? For example, SSL requires certificates, and your Python code doesn't seem to be using any, therefore it will fail to authenticate and "timeout" – OneCricketeer Jun 16 '22 at 13:21
  • @OneCricketeer I'm not sure that I have any SSL certifications. I've tried to add ```listeners``` and ```advertised.listeners``` but it doesn't seem to be working. Here are some of my ```servers.properties```: ```listeners=PLAINTEXT://:9092 listeners=PLAINTEXT://:9098``` ```advertised.listeners=PLAINTEXT://boot-name.c1.kafka-serverless.reagion-name.amazonaws.com:9098 advertised.host.name=boot-name.c1.kafka-serverless.reagion-name.amazonaws.com:9098``` – Nina Jun 19 '22 at 07:56
  • how did you finally fix it? – Yash Kumar Verma Sep 08 '22 at 21:24
  • 1
    @YashKumarVerma I didn't. I created a MSK cluster instead of MSK Serverless and managed to set it up there with SSL configurations – Nina Sep 09 '22 at 11:59
  • I have a MSK cluster and the only configuration I've done is to autocreatetopics, do I need to configure something additional? – Yash Kumar Verma Sep 10 '22 at 17:04
  • @YashKumarVerma I had to add permissions to the specific topic for the user that I was using to be able to produce and consume data from this topic – Nina Sep 11 '22 at 10:54
  • what does a user mean here? – Yash Kumar Verma Sep 11 '22 at 13:12

0 Answers0