Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
5
votes
2 answers

Can I write to an AWS MSK Kafka cluster from a Lambda function?

I need to pick up messages from AWS SES and publish them to an AWS MSK (Kafka) topic. Is this possible? My current understanding is that all MSK clients must live in the same VPC. How would I configure a Lambda function that appropriately?
Patrick
  • 51
  • 1
  • 4
4
votes
0 answers

Unable to communicate with glue schema registry using iam roles for service accounts (IRSA)

we use IAM roles for service accounts in EKS for secure resource access. Recently we are trying to adopt to MSK and glue schema registry. sdk we are using is aws-glue-schema-registry. while IRSA works as expected with other services like secret…
Ahsan Naseem
  • 1,046
  • 1
  • 19
  • 38
4
votes
0 answers

How to delete unused topics?

In our design, Kafka topics are created on the fly and used for a short period of time. While new topics are created and used, previously created topics may go out of use. There is a need to periodically delete unused topics. Say for simplicity, we…
Abhijat Alok
  • 41
  • 3
4
votes
1 answer

How to delete a Worker and a plugin on AWS MSK Connect?

Can't find a way to delete a worker configuration and a custom plugin in Amazon MSK service UI.
floating_hammer
  • 43
  • 6
4
votes
1 answer

How connect to MSK cluster from EKS cluster

I am having difficulties connecting to my MSK cluster from my EKS cluster even though both clusters share the same VPC and the same subnets. The security group used by the MSK cluster has the following inbound rules type protocol port…
Felipe Lujan
  • 61
  • 3
4
votes
1 answer

Making AWS MSK public using NLB and IAM authentication - Hostname verification failed

We are working on getting Amazon MSK (Kafka) working with IAM authentication & thereafter making it publicly accessible by DNS using changes in the aws kafka advertised listeners. To make this happen we are following the same infrastructure plan as…
Skinkpajen
  • 91
  • 5
4
votes
1 answer

How to connect to MSK with SASL/SCRAM using Java?

I'm trying to figure out how to create a working Kafka producer against an AWS MSK cluster. The cluster has TLS and SASL/SCRAM enabled. This is my first time using Kafka and the documentation on this is rather slim. The code that I currently have,…
eof
  • 413
  • 4
  • 14
4
votes
0 answers

SSL Authentication for Managed Kafka AWS (MSK) broker validation failure

I am trying to create an Kafka client app (both producer and consumer) using AWS managed Kafka instance (MSK). Also broker to broker communication and client to broker communication is configured as secure via TLS in cluster. CA is AWS Private CA as…
Vikash Mishra
  • 123
  • 2
  • 10
4
votes
4 answers

AWS MSK with IAM - Timeout Exception

We are using Aws MSK, recently they announce the AWS MSK IAM AUTH(https://github.com/aws/aws-msk-iam-auth) When i tried to use it with the bellow configuration on my clients i got the exception: org.apache.kafka.common.errors.TimeoutException:…
Lucas Gazire
  • 301
  • 5
  • 16
4
votes
2 answers

Failing to connect to Kafka on MSK

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python. I wrote this shortcode: from kafka import KafkaProducer import json producer = KafkaProducer( bootstrap_servers=['host1:9092', 'host2:9092'], …
Sean Goldfarb
  • 329
  • 6
  • 22
4
votes
4 answers

Access to amazon msk cluster locally

I have an amazon msk cluster (kafka) in a vpc and I can access it through a client (EC2 instance) that is in the same vpc. my question is .. can i access my cluster from a client that is on my local computer? How can I do this configuration? I tried…
CARLV
  • 71
  • 1
  • 4
4
votes
3 answers

Can you scale DOWN the number of Kafka brokers in an Amazon MSK cluster?

Quick question for those who have used Amazon's MSK (Managed Streaming for Kafka): The docs/FAQ make it clear that you can easily scale up the number of brokers in an MSK cluster. Can you just as easily scale the number back down? I'm not talking…
lietuviuHimnas
  • 135
  • 1
  • 8
3
votes
1 answer

Exposing AWS MSK cluster using own domain name

I am facing an issue with exposing AWS MSK stack - to be more precisely: case of our development process is to deliver for a client, static, publicly accessible domains for MSK brokers (it is a MUST HAVE, because we are including SSL certificates in…
msmells
  • 31
  • 2
3
votes
2 answers

AWS MSK Connect w/ MSSQL Debezium connector fails with disconnect

I am trying to setup a mssql debezium connector with AWS MSK Connect but keep getting the following error messages: Connector error log: [Worker-0a949760f6b805d4f] [2023-02-15 19:57:56,122] WARN [src-connector-014|task-0] [Consumer…
Ethan Davis
  • 93
  • 1
  • 13
3
votes
1 answer

How do I terraform Amazon MSK topics?

Can I terraform using the aws_msk_configuration the topics and their partition counts and other properties so I can disable automatic topic creation on the broker?
Archimedes Trajano
  • 35,625
  • 19
  • 175
  • 265
1
2
3
…
24 25