Questions tagged [aws-msk-connect]
36 questions
0
votes
0 answers
Kinesis Confluent Connector not reaching MSK Cluster
I am trying to add a Kinesis connector to connect my Kinesis Stream with my AWS MSK Cluster, so that I can send messages from the Kinesis to a certain topic in the Kafka cluster.
This is the connector I am trying to add:…
Homer
- 85
- 7
0
votes
1 answer
How to handle nested arrays of struct in Kafka JDBC Sink Connector
I'm trying to handle nested arrays of struct via my Kafka Connect JDBC Sink Connector while writing records to Postgres.
Im using Kafka Connect (MSK Connect) build in AWS MSK. (I cannot use KSQL or Kafka Streams right now, or can i? run kafka…
repcak
- 113
- 8
0
votes
0 answers
Reading AVRO messages fails for AWS glue schema registry deserializer
I have built a kafka pipeline using MSK, MSK Connect, Debezium Postgres Source connector and AWS Glue schema registry.
On producer side, I am able to publish AVRO records with schemas in Glue schema registry, I am using aws glue schema registry…
Pragya Rai
- 23
- 4
0
votes
1 answer
Using multiple topics in AWS MSK Connect
I created a AWS MSK Connector with below configuration. The export to S3 bucket was successful.
Now how do I use wild card configuration for a stack with like 1000 plus topics in MSK cluster. As you can see below I cannot include names of each topic…
Piyush Patil
- 14,512
- 6
- 35
- 54
0
votes
1 answer
Mongodb-kafka connector failing while connecting MSK to Mongodb
Im trying to connect MongoDB with AWS Managed kafka using the source connector (downloaded from Confluent Hub), but while creating AWS connector im getting the error message:
Code: InvalidInput.InvalidConnectorConfiguration Message: The connector…
A Rafay
- 1
0
votes
1 answer
AWS MSK Connect how to install Rest API
I have created multiple AWS MSK Connectors using Terraform, I came to know we can also use Rest API to update capacity configurations. Everywhere it says http://localhost:8083 but not sure how to use API. My MSK Cluster is running 9094 so it is…
SatyaG
- 25
- 6
0
votes
0 answers
How i connect aws-msk cluster with node js using kafka.js
I am trying to connect aws-msk cluster to our node js application running on azure for publish and suscribe data
any idea for this problem
Sada Mandal
- 1
- 1
0
votes
1 answer
MSK Connector fails to be created
I am trying to create an MSK connector using kafka-connect-aws-s3-kafka-2-8-4.0.0.zip, but it is stuck in ** Creating...** status and it fails with an unknown error.
So a bit about my connector config:
I am trying to deploy this Connector in order…
MBA
- 353
- 2
- 11
0
votes
0 answers
AWS MSK MongoDB Sink Connector with IAM Auth enabled
I have created a AWS MSK Cluster with both IAM and SCRAM authentication enabled. When trying to create a MongoDB sink connector that writes messages from kafka topics to MongoDB, the connector creation fails with a message saying
There is an issue…
junelane
- 101
- 3
- 13
0
votes
1 answer
MSK Java producer/consumer with both key and value in AVRO and using glue schema registry
I am trying to use the MSK connect Mysql CDC connector with both key and value as AVRO schemas using Glue Schema registry (GSR).
When I was doing this using confluent schema registry, the schema name for both key and value will be something like…
Anand K
- 293
- 3
- 12
0
votes
0 answers
why lenses s3 source connector not working?
I am trying to use lenses s3 source connector in aws msk(kafka).
Download kafka-connect-aws-s3-kafka-3-1-4.0.0.zip as a plug-in, save it in S3 and register it.
And the above plug-in was specified and the connector configuration was written as…
jojo
- 3
- 3
0
votes
1 answer
MSK S3 sink not working with Field Partitioner
I am using AWS MSK and msk connect. S3 sink connector is not working properly when I added io.confluent.connect.storage.partitioner.FieldPartitioner Note:without fieldPartitioner s3sink had worked. Other than this stack overflow Question Link I was…
Sreenath
- 91
- 1
- 5
0
votes
1 answer
AWS MSK Connect with IAM - Timeout Exception
I have a setup with a MSK Cluster and an MSK Connect Connector (which is a source connector to a postgres db, but that shouldn't matter here).
During the first step of my setup, it set it all up without authentication. And it worked fine.
Now, as my…
DAndy
- 117
- 11
0
votes
0 answers
Unable to Connect to AWS Glue Schema Registry
I am trying to connect to AWS Glue Schema registry and would like to use WebIdentityTokenFileCredentialsProvider.
As part of it, I tried the following in my custom serializer class ( extends GlueSchemaRegistryKafkaSerializer )
…
Topic_explorer
- 23
- 7
0
votes
0 answers
Error in MSK Kafka Connect sink connector due to Schema Registry
I currently have the following data in my Kafka topic:
{"tran_slip":"00002060","tran_amount":"111.22"}
{"tran_slip":"00000005","tran_amount":"123"}
{"tran_slip":"00000006","tran_amount":"123"}
{"tran_slip":"00000007","tran_amount":"123"}
Since the…
Stephanie Meilak
- 1
- 1