Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
0
votes
1 answer

AWS MSK - Debezium Postgres Connector for AWS RDS - Failed to Connect

I'm currently facing the following issue when using AWS MSK Connector (Debezium Postgres Connector) [Worker-0509fac07b9701a23] [2022-01-19 04:55:28,759] ERROR Failed testing connection for…
hbhargav
  • 1
  • 2
0
votes
1 answer

Creating MSK event source mapping for Lambda function fails

I am using the AWS-CDK to create a stack with an AWS-MSK cluster and a Lambda function which should be triggered, when a new message is available in a specific topic. I already had it working nicely and then I decided to add clientAuthentication and…
ghost23
  • 2,150
  • 1
  • 20
  • 35
0
votes
1 answer

How to measure kafka reBalance duration

is there tool available to measure Kafka re-balancing duration? or check any intermediate status? we have observe a many time, specific consumer get stuck forever during Kafka rebalancing, we never waited to finish.
kus
  • 446
  • 3
  • 7
0
votes
1 answer

Mirror Maker2 not able to connect to target cluster broker

I have two Kafka clusters on AWS MSK (in same environment and region). I have a KafkaConnect cluster setup on the destination cluster and have setup a mirror maker connector to run. The submission of the connector is fine and there are no…
guru
  • 409
  • 4
  • 21
0
votes
0 answers

How do I configure a consumer to check more than one schema when listening to multiple topics?

I'm working on a project for a large company with millions of users. We are attempting to convert their REST based architecture to an event based architecture. The current architecture involves a service, we'll call it Service-A, that makes 7 REST…
user3236794
  • 578
  • 1
  • 6
  • 16
0
votes
1 answer

How to automate the AWS MSK Topics Creation Without using EC2 Instance

We've set up the MSK Cluster, VPC, and Security Group in AWS. We want to create an AWS MSK topics without using EC2 instance. Is it possible to automate this? or any other way to do this?
Manjunath Davanam
  • 195
  • 1
  • 11
0
votes
1 answer

Extract data from terraform output to use in Secrets Manager

I have a terraform script that builds a MSK cluster, I have outputted the zookeeper & broker information and I would like to add this data to a AWS secret's manager secret. The problem is they are comma separated. This is the output in the pipeline …
chris-ellis
  • 21
  • 4
0
votes
1 answer

AWS DMS unable to write to MSK target

Note: self-answered question, because Google didn't shed any light on the problem. I have configured a Managed Streaming for Kafka target for AWS Data Migration Service, but the migration job fails. Looking at the logs, I see…
kdgregory
  • 38,754
  • 10
  • 77
  • 102
0
votes
1 answer

Handle rotation of password in spring cloud stream consumers when connected to MSK kafka

I have a Kafka consumer written using Spring Cloud Stream which connects to Kafka deployed on Amazon MSK cluster with secret manager for periodic password rotation. Problem: Do I need to re-instantiate the application on every password rotation? If…
Hari Pillai
  • 1
0
votes
3 answers

Kafka consumer using AWS_MSK_IAM ClassCastException error

I have MSK running on AWS and I'd like to consume information using AWS_MSK_IAM authentication. My MSK is properly configured and I can consume the information using Kafka CLI with the following command: ../bin/kafka-console-consumer.sh…
Danilo Cairolli
  • 51
  • 4
0
votes
2 answers

How to connect to AWS MSK from Debezium connector using username and password

Here i'm trying to use Debezium Connector to Read data from RDS and publish it to AWS MSK. All i could see is using AWS IAM based authentication everywhere but unfortunately my cloud engineering team is not willing to add IAM but provided basic auth…
Work Work
  • 29
  • 4
0
votes
1 answer

reason for adding console consumer in deny list in burrow configuration

I recently started using burrow and for burrow configuration, I see below configuration for group-deny list at lots of places including AWS MSK documentation: https://docs.aws.amazon.com/msk/latest/developerguide/consumer-lag.html But I cannot find…
Vikash Mishra
  • 123
  • 2
  • 10
0
votes
1 answer

AWS MSK Trigger - Lambda (consumer) running infinitely

I am working on a notification service which is built on AWS infra and uses MSK, lambda and SES. The lambda is written in Nodejs for which the trigger is an MSK topic. Now the weird thing about this lambda is its getting invoked continuously even…
arpit
  • 51
  • 1
  • 3
0
votes
2 answers

AWS MSK Default enable.auto.commit setting

Does AWS' Managed Streaming for Apach Kafka (MSK) have enable.auto.commit set to true by default? I't not showing up in the docs, and I want to make sure that my consumers are properly committing offsets. Or does MSK follow the Kafka process cited…
Todd Holmberg
  • 450
  • 1
  • 4
  • 13
0
votes
1 answer

MSK, IAM, and Kafka Java Api

So for some reason I can't get my connections just right with MSK via the Kafka Java API. I can get producers/consumers to work with MSK using conduktor and Kafka CLI tools. However when I try to hook up my Scala code I can't get it to work. So I am…
l33tHax0r
  • 1,384
  • 1
  • 15
  • 31
1 2 3
…
24 25