Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
2
votes
1 answer

Not able to connect AWS MSK connect cluster to MSK cluster

I am unable to use my msk cluster when creating the connect app using aws msk connect cluster. I get error You must choose an authentication method from your selected cluster. I have enabled IAM based authentication on the cluster along with…
Naresh
  • 187
  • 5
  • 11
2
votes
1 answer

what does "Client subnets" mean in Amazon Managed Streaming for Apache Kafka?

The AWS MSK documentation said: clientSubnets: The list of subnets to connect to in the client virtual private cloud (VPC). Amazon creates elastic network interfaces inside these subnets. Client applications use elastic network interfaces to…
Ryan Lyu
  • 4,180
  • 5
  • 35
  • 51
2
votes
1 answer

Connecting to AWS MSK Kafka from my laptop: NoBrokersAvailable

I am having trouble connecting to AWS MSK Kafka (with with Access Control Method configured to None) from my laptop, error is: NoBrokersAvailable I am using plaintext conn over port 9092: from kafka import KafkaProducer kafka_brokers = […
user3761555
  • 851
  • 10
  • 21
2
votes
3 answers

How to monitor disk space usage for Kafka Brokers in AWS MSK cluster

We need to Monitor disk space usage for Kafka Brokers running in AWS MSK cluster. There're several metrices emitted by Kafka which can be used to monitor various aspects. But I was unable to find any specific metric that monitors "Disk Usage" for…
Himanshu Singh
  • 199
  • 3
  • 15
2
votes
1 answer

Kafka Admins timeout

I have an AWS MSK cluster up and running. Connected to it and ran this command to create a test topic called topicoteste usr/local/kafka_2.13-2.5.0/bin/kafka-topics --create --bootstrap-server BOOTSTRAP_STRING_HERE --partitions 1…
Pedro Cavalcante
  • 414
  • 4
  • 14
2
votes
0 answers

AWS MSK setting advertised listeners

I am trying to set the advertised.listeners config of my MSK cluster. However, I am getting the following error back: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners I am not sure what this…
prats
  • 225
  • 5
  • 15
2
votes
0 answers

AWS Lambda with MSK topic as event source not able to read Kafka messages

I have created an AWS MSK cluster and have tested the set up by running a EC2 Producer/Consumer instance in the same VPC.I have used the same VPC and subnets to create an AWS lambda with a MSK cluster topic as an event source.All my subnets are…
Sam
  • 295
  • 2
  • 9
  • 24
2
votes
1 answer

How do you use a nodejs Lambda in AWS as a producer to send messages to MSK topic without creating EC2 client server?

I am trying to create a Lambda in AWS that serves as a producer to an MSK topic. All the AWS docs say to create a new EC2 instance, but as my Lambda is in the same VPC I feel like this should work. I am very new to this and I notice my log statement…
drumurr
  • 33
  • 1
  • 4
2
votes
1 answer

Can I consume an AWS MSK kafka topic from a Lambda function?

I thought it would be trivial to use a lambda to consume messages coming to a topic in a AWS MSK Kafka cluster, but I could not find a way to do it from the AWS docs. Is it possible in some way?
Carlo Pires
  • 4,606
  • 7
  • 32
  • 32
2
votes
1 answer

Creating MSK topic using python script and boto3

I want to develop a python script to create topics in MSK using AWS SDK boto3. I don't see any methods to create a topic in Kafka.client. I can use kafka-python but wanted to use AWS SDK or CDK Please suggest.
dvlpr
  • 311
  • 3
  • 17
2
votes
2 answers

Is there a way to dump Amazon MSK Topic to S3 directly?

I have planned to used Amazon MSK and i want to dump consumer logs to S3 . But i don't see any options. Do i need to write my own consumer or is there a way to consume Amazon MSK consumer output to s3 directly ?
Navin Kumar
  • 150
  • 2
  • 10
2
votes
1 answer

Connection problem with msk using spring kafka

So here on the aws instance we have tried to connect spring kafka with msk where it works with local kafka. next there is no ssl . I tried telnet its running. But while running the application this exception comes. Our cluster of msk is in same…
Tanmay Naik
  • 586
  • 1
  • 4
  • 16
2
votes
4 answers

Provide hostname to Schema Registry instances running within ECS

I am building a redundant Schema Registry hosted in Amazon for our MSK Kafka Cluster by using an ECS cluster. The SchemaRegistry TaskDefinition needs to define a hostname which is unique to each Task when running. SchemaRegistryTaskDefinition: …
Syntax
  • 2,155
  • 2
  • 23
  • 34
2
votes
1 answer

Handling kafka clients updates in kubernetes

I have a Kafka cluster running on AWS MSK with Kafka producer and consumer go clients running in kubernetes. The producer is responsible for sending the stream of data to Kafka. I need help solving the following problems: Let's say, there is some…
Piyush Kumar
  • 21
  • 1
1
vote
0 answers

KafkaJS throwing "KafkaJSNumberOfRetriesExceeded: This is not the correct coordinator for this group" with MSK

I have a nestJS application with KafkaJS deployed in an EKS cluster. I am using Amazon MSK as my kafka cluster with three brokers and SASL SCRAM SHA512 Authentication. The Kafka cluster has default configuration. Here is my consumer…
Sampath
  • 11
  • 2
1 2 3
…
24 25