Stack Overflow
Stack Overflow

Questions tagged [aws-msk-connect]

36 questions
4
votes
1 answer

How to delete a Worker and a plugin on AWS MSK Connect?

Can't find a way to delete a worker configuration and a custom plugin in Amazon MSK service UI.
floating_hammer
  • 43
  • 6
3
votes
1 answer

Confluent S3 Sink Connector for Kafka unable to connect to S3 using MSK / MSK Connect

I'm attempting to store my Kafka messages to S3 using Apache MSK / MSK Connect and the Confluent S3 Sink Connector. I'm experiencing a timeout error when the connector tries to reach S3. To rule out a permissions issue, I've given the connector role…
NRaf
  • 7,407
  • 13
  • 52
  • 91
3
votes
1 answer

Connect to MongoDB atlas within AWS VPC using AWS MSK connector

I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services. I created an AWS MSK cluster within the VPC and created an AWS MSK connect…
Jerin A Mathews
  • 8,572
  • 4
  • 26
  • 49
2
votes
1 answer

How to connect to AWS MSK Cluster from C# using Identity and Access Management (IAM) authorization

I have an AWS MSK (Kafka) Broker. It has been setup for public access authenticating with IAM. I know the broker settings are correct, I have been using https://github.com/aws/aws-msk-iam-auth to successfully connect to the broker as both a consumer…
Rob Gorman
  • 3,502
  • 5
  • 28
  • 45
2
votes
5 answers

Amazon MSK failed to associate 1 secret for cluster. The provided secret has an invalid schema

We want to set up Username and password authentication with AWS Secrets Manager as per the documentation. We created a cluster in MSK Created a secret as well with name AmazonMSK_testmsk2 and with key as Password and Value as { "username": "alice",…
codeaprendiz
  • 2,703
  • 1
  • 25
  • 49
1
vote
0 answers

Debezium resulting in high transaction log disk usage

I am trying to setup CDC based replication for different databases (Amazon Postgres RDS) using multiple Debezium Kafka Connectors (using MSK Connector). With addition of more connectors, the replication is creating issues. The Oldest Replica Slot…
Keshav Rathi
  • 11
  • 3
1
vote
0 answers

Connecting Mongo Atlas with AWS MSK without VPC

I was trying to connect Mongo Atlas from Amazon AWS MSK Connect , but not able to connect it. I had provided the mongo connection uri , when creating a MSK connector. There are solutions in which VPC and Private Endpoints are being used . but is…
Ritik
  • 45
  • 1
  • 5
1
vote
0 answers

MSK connector to handshake with a third party hosted cluster using Kafka SSL or SASL Authentication

I have created a kinesis source connector which is supposed to pull data from the kinesis stream and push the same data to kafka topic. The use case here is connector will be deployed on Amazon MSK connect and where as the topic is created on some…
Yogesh Katkar
  • 133
  • 10
1
vote
0 answers

AWS MSK connection issue using IAM Role authentication from Logstash

my logstash configuration is i am using aws-msk-auth plugin kafka { codec => "json" topic_id => "my_topic" bootstrap_servers => "my_iamBroder:9098" security_protocol => "SASL_SSL" sasl_mechanism =>…
Shihabudheen K M
  • 1,347
  • 1
  • 13
  • 19
1
vote
1 answer

Kafka connector Failing to connect AWS MSK

I am trying to configure MSK connect in AWS and the below is the configuration. INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235) [Worker-02003b81ffe0ee9c3]…
Vishnu
  • 11
  • 1
  • 3
1
vote
0 answers

MongoDB Debezium Fails to connect due to ssl handshake failure

I'm running a MongoDB Debezium Kafka Connector on AWS MSK, and the connector goes to the failed status with this error on the MongoDB server Error receiving request from client: SSLHandshakeFailed: The server is configured to only allow SSL…
taraf
  • 777
  • 2
  • 10
  • 28
1
vote
1 answer

Using Glue schema registry with MSK Connector

I've been trying to create an MSK Connector and use Glue schema registry with it. The configuration is as…
Rohit
  • 35
  • 6
1
vote
2 answers

Is there a way to update connector configuration using MSK-connect API?

I have an s3 connector deployed on MSK Connect, and a repository on github with the json connector configuration file. I'd like to update the connectors configuration on demand via MSK's REST API. I've checked the API documentation, but it seems…
omer
  • 1,242
  • 4
  • 18
  • 45
1
vote
0 answers

How to fix msk connector glue registry timeout error?

I have an MSK cluster setup on 2 public subnets in eu-west-1 region. I have a topic named 'awsuseravro1' and Glue Schema Registry with name 'msk-registry' and schema name same as the topic name. I have created an MSK connector with a custom plugin…
Konrad Paniec
  • 11
  • 1
0
votes
0 answers

MSK Connect can't connect community mongodb and error UnknownHostException

On AWS, I had EC2 mongodb cluster on a VPC and tried to use MSKConnect to capture change stream and publish to MSK on another VPC. Both VPCs can reach each others via TransitG​W. I also had EC2 on the same subnet as MSK. The subnet was allowed to…
Pakorn K
  • 111
  • 1
  • 11
1
2 3