For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
Questions tagged [aws-msk]
371 questions
2
votes
0 answers
Kafka Connector : Encountered change event for table table name whose schema isn't known to this connector
We are facing issue with our new connector, we want to capture a change event for mysql table, but we are noticing the error with trace
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This…
Vandit Shah
- 21
- 2
2
votes
2 answers
Concurrency of Lambda not increasing as expected
We've configured an MSK (kafka) event source as the trigger for our Lambda function. Even though the offset lag is increasing the lambda concurrency is limited to 4-5 almost all the time as can be seen in the graph below. The configuration used for…
pkgajulapalli
- 1,066
- 3
- 20
- 44
2
votes
1 answer
How to connect to AWS MSK Cluster from C# using Identity and Access Management (IAM) authorization
I have an AWS MSK (Kafka) Broker. It has been setup for public access authenticating with IAM. I know the broker settings are correct, I have been using https://github.com/aws/aws-msk-iam-auth to successfully connect to the broker as both a consumer…
Rob Gorman
- 3,502
- 5
- 28
- 45
2
votes
0 answers
When trying to list the consumer groups, i get timeout exception
i need to list my consumers in my bootstrap server and for that i'm using below command.
.bin/kafka-consumer-groups.sh --bootstrap-server b-2.amazonaws.com:9098,b-3.amazonaws.com:9098,b-1.amazonaws.com:9098 --list --command-config…
Dushan
- 1,365
- 20
- 26
2
votes
1 answer
Kafka MSK - a configuration of high fetch.max.wait.ms and fetch.min.bytes is behaving unexpectedly
I have a Kafka consumer running on a Spring application.
I am trying to config the consumer with fetch.max.wait.ms and fetch.min.bytes.
I would like the consumer to wait until there are 15000000 bytes of messages or 1 minute has…
mosh
- 404
- 2
- 8
- 16
2
votes
1 answer
Kafka Mirrormaker 2 replication from latest offset instead of earliest
I have a problem with replication of topics from cluster A to cluster B with Mirrormaker 2. I want to replicate these topics from latest offset instead of replicating them from the beginning. But when I set auto.offset.reset=latest nothing happens…
Scourrge
- 83
- 7
2
votes
5 answers
Amazon MSK failed to associate 1 secret for cluster. The provided secret has an invalid schema
We want to set up Username and password authentication with AWS Secrets Manager as per the documentation.
We created a cluster in MSK
Created a secret as well with name AmazonMSK_testmsk2 and with key as Password and Value as { "username": "alice",…
codeaprendiz
- 2,703
- 1
- 25
- 49
2
votes
0 answers
Data Learn (AWS MSK with Google Dataflow connection issue)
My Google Dataflow Job is keep getting failed while creating "Kafka-Bigquery" dataflow with AWS MSK public access bootstrap server. Is there any way to solve this issue?
I tried accessing public access bootstrap server by disabling unauthenticated…
Anas Vakyathodi
- 29
- 3
2
votes
0 answers
Mirror-maker 2 is not copying old data
I have 2 MSK kafka clusters A and B . I want to copy data from A to B using MM2 . It copies new data but older data is not getting copied from a to B .
MM2 configuration file -
A.bootstrap.servers=
B.bootstrap.servers=
Pankaj Yelulkar
- 21
- 3
2
votes
0 answers
How to use Amazon MSK in C# code to publish a message to Kafka
I am creating a c# application to publish a message to Kafka. In the current version of my application I set up cluster locally using docker (confluentinc-cp and confluentinc-zookeeper). However to run Kafka efficiently, we decided to use Managed…
PurpleGreen
- 39
- 8
2
votes
2 answers
What kafka-connect configs influence number of TCP connections to kafka cluster?
What are the settings that affect the number of TCP connections made to kafka? Background is that MSK IAM has a throttle limit.
Some things i can think of:
max.tasks
number of partitions
number of brokers
replication factor
tooptoop4
- 234
- 3
- 15
- 45
2
votes
2 answers
Using Confluent Schema Registry with MSK
Is it possible to integrate Confluent Schema Registry with AWS MSK?
If you have done this before, can you please provide some pointers / blogs you followed to achieve it?
guru
- 409
- 4
- 21
2
votes
1 answer
AWS MSK trigger for Lambda `Batch Window` like configuration
What is the batching window used for MSK as an event source? It seems there is a support for Batch Window and MaximumBatchingWindowInSeconds for SQS as an event source, however how to provide the same params for lambda for MSK…
vin
- 960
- 2
- 14
- 28
2
votes
1 answer
Spring Boot AWS MSK Configuration
To consume messages from Kafka, I am using the below configuration in my local. It is working fine.
spring.kafka.bootstrap-servers: localhost:9092
spring.kafka.consumer.key-deserializer:…
user1346346
- 195
- 2
- 16
2
votes
1 answer
how to stream data from AWS MSK (kafka) to snowflake using MSK connect
I'm trying to set up a MSK connector for snowflake and i could hardly see any documentation on how to do it. Unfortunately AWS support person also referred me to use snowflake documentation page.
By following this i can create an EC2 instance and…
Work Work
- 29
- 4