For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.
Questions tagged [aws-msk]
371 questions
0
votes
1 answer
AWS DMS with AWS MSK(Kafka) CDC transactional changes
I'm going to use AWS Database Migration Service (DMS) with AWS MSK(Kafka).
I'd like to send all changes within the same transaction into the same partition of Kafka topic - in order to guarantee correct message order(reference integrity)
For this…
alexanoid
- 24,051
- 54
- 210
- 410
0
votes
2 answers
How to increase Kafka Consumers no
Say for example -
Kafka topic name - topic_X
topic_X has 500 partitions in the Kafka Broker
Now we have set 500 consumers for the Consumer_X group to process each partitions.
where to Run these 500 consumers ?
In a single machine 500 thread ? Is…
omg
- 31
- 3
0
votes
1 answer
Using Apache Kafka in place of SQS
I have an application that uses AWS SQS with Lambda to process the messages pushed on the Queue. The Lambda keeps on polling the Queue, and when a new message appears it process the message.
For this scenario, is it possible to replace the SQS with…
gtango
- 21
- 4
0
votes
1 answer
Kafka: Topic list discrepancy between zookeeper and bootstrap-server
On a couple of my clusters I'm seeing a discrepancy between the list of topics returned by zookeeper as compared to the broker i.e the following commands return different (fewer in the case of the broker) results
kafka-topics.sh --zookeeper…
JustCatchingRye
- 11
- 3
0
votes
1 answer
Amazon MSK cluster on private VPC access with TLS enabled
I have an AWS MSK kafka cluster on a private VPC. I have set up a bastion to connect to this cluster and can successfully produce and consume messages from the bastion using kafka console commands and using the JVM keystore for SSL auth.
I am trying…
aamirnshah
- 59
- 5
0
votes
1 answer
What is the difference between Kafka Cluster and Kafka Broker?
Has Kafka cluster and Kafka broker the same meaning?
I know cluster has multiple brokers (Is this wrong?).
But when I write code to produce messages, I find awkward option.
props.put("bootstrap.servers", "kafka001:9092, kafka002:9092,…
JoonT
- 1,106
- 1
- 13
- 29
0
votes
1 answer
Kafka on Kubernetes
We have 12 API's deployed on a cluster and we are using Kafka which are deployed on 3 EC2 instances. Should I add the Kafka Servers in K8s too or should I keep it the same? Or should I start using AWS MSK?
Still Experimenting so any suggestions or…
0
votes
1 answer
Spring Cloud Stream with AWS MSK - How can I use and deploy on AWS?
I have been using Spring Cloud Stream with Kafka for a while now. I have a sample SCS-Producer which pumps out data to an output topic.
Below is my producer code:-
@EnableBinding(Source.class)
public class SampleProducer {
…
Kenny Weeler
- 153
- 1
- 3
- 15
0
votes
1 answer
Event data streaming applications with apache kafka in local env and move it to AWS Kafka for pre-prod / prod
Is it possible to develop event data streaming applications with apache kafka locally in my laptop using kafka docker and then deploy the application to AWS msk (kafka) so that cost can be saved by using only for pre-prod and production.
0
votes
0 answers
Configure schema registry to only use "CLIENT_SECURE" protcol mapping to connect to broker list returned by zookeeper
I am running a Kafka / Zk cluster in AWS MSK, and am connecting schema registry to it.
If I specify the broker list in kafkastore.bootstrap.servers, it connects fine (using protocol SSL). However, if I remove bootstrap server config and switch to a…
Brett
- 5,690
- 6
- 36
- 63
0
votes
0 answers
Mulesoft Kafka connector and Amazon MSK
I am trying to connect Mulesoft to Amazon MSK using the Kafka connector. I followed the steps from amazon https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html but not sure if I need to create that EC2 instance.. is a…
Santos
- 177
- 1
- 1
- 15
0
votes
2 answers
Not able to start Kafka connect for Elastic search in Distributed mode
I am trying to start Kafka connect in distributed mode even in standalone also i am not able to proceed
This is my elastic search sink properties…
Sudarshan kumar
- 1,503
- 4
- 36
- 83
0
votes
0 answers
Connection to MSK using spring kafka
Hi I am using aws MSK and trying to connecting to it using spring kafka client. While sending message to kafka cluster from my code, I get following error-
2020-01-27 21:49:17.141 ERROR 4176 --- [nio-9000-exec-4]…
Aashish Katta
- 1,174
- 3
- 13
- 22
0
votes
1 answer
Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
I have MSK running on aws and i am able to send records in and out from MSK .
I just wanted to use Kafka connect so that records coming into MSK will go to Elastic Search .
I have done below things but i am not sure if my connector is working…
Atharv Thakur
- 671
- 3
- 21
- 39
0
votes
1 answer
Streaming data from AWS DMS to AWS MSK using lambda function
So i am new to Kafka and MSK but trying to explore thing server less on AWS cloud .
My use case
AWS DMS-->KAFKA(MSK)-->ElastciSearch/DynamoDB and S3.
i am looking to do this using lambda function.
But recently i came to know end to end can not be…
Sudarshan kumar
- 1,503
- 4
- 36
- 83