For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
Questions tagged [aws-msk]
371 questions
0
votes
0 answers
MSK How to mandate both SASL & SSL protocols while validating the client to connect msk broker nodes?
In MSK to communicate with brokers by using TLS(SSL) we have used 9094 port and able to communicate MSK cluster with the required keystore and truststore configurations.
And to communicate with brokers by using SASL_SCRAM we have used 9096 port and…
Rajashekhar Meesala
- 313
- 1
- 14
0
votes
0 answers
Unable to Connect to AWS Glue Schema Registry
I am trying to connect to AWS Glue Schema registry and would like to use WebIdentityTokenFileCredentialsProvider.
As part of it, I tried the following in my custom serializer class ( extends GlueSchemaRegistryKafkaSerializer )
…
Topic_explorer
- 23
- 7
0
votes
0 answers
Unable to write files to S3 using MSK S3 sink connector
I have a Kafka instance running on an AWS EC2 machine and acting as a producer to my AWS MSK Cluster. For writing the data to S3 bucket I have created an AWS MSK Connector using below…
Prashant Vikram Singh
- 101
- 1
- 10
0
votes
2 answers
How to consume Avro Serialized messages from AWS MSK via Apache Beam
PCollection> kafkaRecordPCollection =
pipeline.apply(
KafkaIO.read()
.withBootstrapServers("bootstrap-server")
.withTopic("topic")
…
0
votes
0 answers
Error in MSK Kafka Connect sink connector due to Schema Registry
I currently have the following data in my Kafka topic:
{"tran_slip":"00002060","tran_amount":"111.22"}
{"tran_slip":"00000005","tran_amount":"123"}
{"tran_slip":"00000006","tran_amount":"123"}
{"tran_slip":"00000007","tran_amount":"123"}
Since the…
Stephanie Meilak
- 1
- 1
0
votes
1 answer
AWS MSK gives warning
we are working on spring boot application and we have implemented MSK AWS (Kafka) , producer and consumer communication is working as expected but once I checked logs it is giving warning as below.
Connection to node -1 (localhost/127.0.0.1:9092)…
Kapil Devmurari
- 159
- 3
- 11
0
votes
1 answer
How to create Connector CustomPlugin using AWS_MSK CDK?
I am using CfnConnector (as the below documentation reference link states) to create a MSK connector and it requires a customPluginArn. How am I able to create a custom plugin using the CDK, so i am able to get the CustomPluginArn and the Revision…
D.B
- 4,009
- 14
- 46
- 83
0
votes
0 answers
intermittent issue with kafka (aws msk) consumer
We are facing a strange issue in only one of our environment (with same consumer app).
Basically, it is observed that suddenly a lag starts to build up with only one of the topics on kafka broker (it has multiple topics), with 10 consumer members…
Arpit S
- 137
- 2
- 10
0
votes
1 answer
Amazon MSK pricing for EBS storage
In Amazon MSK, the pricing documentation has mentioned that - You pay for the amount of storage you provision in your cluster.
But while creating the msk cluster, we have defined initial volume size per broker is 100gb, And our cluster has utilized…
Rajashekhar Meesala
- 313
- 1
- 14
0
votes
1 answer
Automating MSK details retrival with Makefile and JSON
I'm currently working on a cloudformation script that requires a lot of MSK configuration details to run. I am working on a makefile that runs the command
@aws kafka list-clusters
This returns a json-like structure that can be found here . Most of…
veila
- 83
- 8
0
votes
0 answers
Open firewall between IBM MQ to AWS account
I have a use case where data is populating from IBM MQ(Source) and trying to connect AWS MSK cluster. I created AWS cluster and I have the IBM MQ config details to create a connector. I wanted to open a secure firewall between AWS and IBM MQ.
Can…
Data Enthusiast
- 1
- 2
0
votes
1 answer
service not consume message from all topic consumer group on AWS Kafka Cluster
I am very new to AWS and Kafka Cluster.
We have NodeJs service which connects to two different Kafka cluster. We have multiple instances of the consumer service and each instance consume a record from every topic but from last couple of weeks we…
ppb
- 2,299
- 4
- 43
- 75
0
votes
1 answer
How to Run Kafka connect in distributed manner in production area
AS per documentation on confluent https://docs.confluent.io/platform/current/connect/references/restapi.html we can use connect rest api to start/stop/manage kafka connect - just want to understand what is the preferable way when we are deploying…
0
votes
1 answer
Strimzi Kafka ssl validation failed VS AWS MSK ssl passed
I have nodejs app using kafkajs package for connecting to AWS MSK.
We are moving to Strimzi Kafka because we already have a kubernetes cluster and we don't need the MSK anymore.
Until now we were connected with SSL but didn't have to specify any CA…
yershalom
- 786
- 1
- 8
- 19
0
votes
1 answer
Java Lambda function connnect MSK cluster timeout
I have created a java lambda function within same vpc, security group of MSK cluster. But when lambda execute the code, in cloudwatch:
org.apache.kafka.common.errors.TimeoutException
My java creating topic code like this:
public String…
Will Zefeng Qiu
- 41
- 1