For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
Questions tagged [aws-msk]
371 questions
1
vote
0 answers
AWS MSK connector with kafka-connect-elasticsearch error ERROR Failed to create client to verify connection
I try to connect my aws msk which is in private vpc with msk connector to my cloud.elastic.io elasticsearch. MSK cluster and elastic.io are in same region us-west-2
I use https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch but…
Vahan Hovsepyan
- 11
- 1
1
vote
1 answer
Log more data from Kafka Source Connector
I installed the Kinesis Connector plugin in AWS, to connect my Kinesis Stream with my MSK cluster. It is working, but the logging that I see in Cloudwatch is not very helpful:
[Worker-07578247b0d45ad42] [2023-08-09 07:51:20,145] INFO…
Homer
- 85
- 7
1
vote
0 answers
Move data from Redis pubsub to Kafka(AWS MSK)
We are building a real time collaborative application using Redis pub-sub and Kafka. We want to move all the data received in the Redis pub-sub(irrespective of the channel) to Kafka and we came across the Redis Connector (Source and Sink) by Redis…
SB Praveen
- 49
- 6
1
vote
0 answers
Unable to connect KEDA to AWS MSK with scaled object
I have created an AWS MSK cluster with 3 brokers. The authentication to the cluster is with IAM role-based authentication and SASL/SCRAM authentication. I already use MSK within my microservices and I send and receive events in the brokers. Now I…
Maor agai
- 221
- 1
- 3
- 11
1
vote
1 answer
Sending data to a specific Kafka paritions using AWS DMS
As per the question,
Is it possible to configure
an AWS DMS instance to one specific parition of an aws kafka
TABLE 1 -> DMS 1 -> Kafka_Topic(parition 0)
TABLE 2 -> DMS 2 -> Kafka_Topic(parition 1)
just asking a question
eeeVillain
- 11
- 2
1
vote
1 answer
Create MSK Kafka topic through Terraform
I am trying to create topics through Terraform, in a AWS MSK Kafka cluster, using the Mongey/kafka provider, but I always get connection refused.
I have the cluster configured as follows:
resource "aws_msk_cluster" "this" {
cluster_name …
Homer
- 85
- 7
1
vote
0 answers
What are the configurations needed to connect spring boot app and AWS MSK?
In local Kafka messages are consuming successfully with the below application configuration
spring:
cloud:
stream:
kafka:
binder:
replicationFactor: 1
auto-create-topics: true
brokers:…
Manoj Kumar
- 21
- 2
1
vote
0 answers
Unable to use Debezium postgres(11) with rds
Unable to setup Debezium Postgres connector using MSK connect on RDS
Trying to set up Postgres Connector with Debezium with Postgres on RDS. The highest previledge granted by AWS is postgres user having rds_superuser role, which doesnot have…
Niteesh Hegde
- 25
- 2
1
vote
0 answers
Debezium resulting in high transaction log disk usage
I am trying to setup CDC based replication for different databases (Amazon Postgres RDS) using multiple Debezium Kafka Connectors (using MSK Connector). With addition of more connectors, the replication is creating issues. The Oldest Replica Slot…
Keshav Rathi
- 11
- 3
1
vote
0 answers
Python: confluent-kafka (librdkafka) producer Timeout error
I've got a 9 nodes kafka cluster hosted on AWS MKS and I'm using confluent-kafka library with python.
While producing to a topic I get too many Timeout errors like:
%5|1684850550.061|REQTMOUT|eb3004f09ce7#producer-1| [thrd:sasl_ssl://broker1.a]:…
JustAnOtherNickname
- 11
- 3
1
vote
0 answers
Got "org.apache.kafka.common.errors.TimeoutException: Topic xxx not present in metadata after 60000 ms" when write from EMR to MSK
Issue
I have a Spark application in Scala in Amazon EMR (version emr-6.10.0). It tries to write data to Kafka in Amazon MSK (Kafka version 3.4.0) through IAM authentication way.
I have created a topic called hm.motor.avro in the Amazon MSK…
Hongbo Miao
- 45,290
- 60
- 174
- 267
1
vote
0 answers
Connecting Mongo Atlas with AWS MSK without VPC
I was trying to connect Mongo Atlas from Amazon AWS MSK Connect , but not able to connect it.
I had provided the mongo connection uri , when creating a MSK connector.
There are solutions in which VPC and Private Endpoints are being used . but is…
Ritik
- 45
- 1
- 5
1
vote
1 answer
Kafka partition huge replica size
We have a managed MSK kafka cluster of 3 brokers.
We have noticed that one of the brokers has a much larger disk usage than others for quite some time.
After some analysis we found out that one of the partition replicas in the __consumer_offsets…
Slava Shpitalny
- 3,965
- 2
- 15
- 22
1
vote
0 answers
`GlueSchemaRegistryDeserializerDataParser` replacement in aws-sdk-java-v2
We're using the GlueSchemaRegistryDeserializerDataParser class from https://github.com/awslabs/aws-glue-schema-registry.
This seems to be from the v1 of the AWS SDK (or am I wrong?)
Is there a replacement in aws-sdk-java-v2?
Jules Ivanic
- 1,579
- 2
- 15
- 28
1
vote
0 answers
How to connect to AWS MSK using Golang?
I have set up an MSK Kafka Cluster with SCRAM security. The VPC allows all incoming and outgoing connections.
I'm attempting to connect to the cluster using the following steps:
mechanism, err := scram.Mechanism(scram.SHA512, "user",…
Raisa A
- 437
- 7
- 21