Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
0
votes
0 answers

Integrating confluent kafka-restproxy with AWS MSK

I am trying to integrate AWS MSK with Confluent provided RestProxy. I hv created the AWS MSK cluster, without authentication for note, and hv deployed the RestProxy provided by confluent on same VPC. After giving the Plaintext zookeeper connection…
Kpierce
  • 1
  • 1
0
votes
1 answer

Unable to update kafka cluster version in AWS MSK

We have written python code to upgrade kafka version in AWS MSK and its giving error .................................................. vaibhavkanchan@vkmacbook python-code % ./update_kafka_version.py /gpn-prime/dev/msk/kafka-cluster 2.6.1 …
vaibhav kanchan
  • 71
  • 6
0
votes
1 answer

How to consume events form from AWS MSK cluster using KafkaConsumer java SDK?

I have created a cluster in AWS MSK and created a topic. I can produce and consume events from the EC2 instance in the same VPC as the cluster. The specific use case is that I want to consume events using Java. The KafkaClient AWS SDK doesn’t have…
sd99
  • 13
  • 4
0
votes
1 answer

AWS MSK, in configuration auto.create.topics.enable=true but not working

We created a new AWS MSK (kafka) cluster We created a new configuration and assigned to that…
Metalik
  • 873
  • 1
  • 10
  • 28
0
votes
1 answer

Send data to elasticsearch from kafka (MSK) AWS

Hello I am currently trying to send data from kafka(MSK)(AWS) to elasticsearch via confluent connector. I have a stream of data coming in from a sql database. When I run the confluent connector it runs for a while sending data to elasticsearch but…
bob Ditusa
  • 21
  • 1
  • 5
0
votes
1 answer

Kafka sends value as a string: How can I deserialize it and turn it into a JSON object with Python

I am trying to consume MSK (managed Amazon Kafka service) messages from a lambda function - MSK is the trigger of my lambda. The producer looks like this: data = {'time': 1611215510000000000, 'tags': {'tag1': 'tagvalue'}, 'fields': {'value':…
Antona
  • 107
  • 2
  • 14
0
votes
1 answer

Are zookeeper nodes in Amazon MSK in different availability zones?

https://docs.aws.amazon.com/es_es/msk/latest/developerguide/what-is-msk.html Reading this documentation I do not clearly see whether zookeeper nodes are within the same subnet and therefore in the same availability zone. I assume that for high…
rey don sancho
  • 453
  • 3
  • 16
0
votes
0 answers

Amazon MSK Spring Boot App Connection Not Working

Hello I'm trying to connect my app into our Amazon MSK Cluster with this code However it always fails to connect into the Amazon MSK cluster kafka: bootstrap-servers: b-1.kafka-cluster.XXX.XXXX.kafka.me-south-1.amazonaws.com:9092 consumer: …
mikster21
  • 1
  • 1
0
votes
1 answer

kafka-connect on ECS timeouts when connecting to MSK

i deployed the kafka-connect docker image(confluentinc/cp-kafka-connect-base:6.0.1) into ECS / fargate, assigned a security group to my ECS service that permits both incoming zooper keeper and kafka bootstrap server traffic (both plain text and TLS)…
austiine
  • 470
  • 1
  • 5
  • 12
0
votes
1 answer

How to find group.id property of a kafka topic in Amazon MSK?

I am trying to consume messages from a Kafka topic in Amazon MSK using Apache NiFi. NiFi ConsumeKafka processor requires group.id property of Kafka.Where or How can I find the group.id property for the MSK cluster?
Tris
  • 63
  • 1
  • 1
  • 8
0
votes
1 answer

Kafka Streams EOS Mode - Informed to Shutdown

I have a Kafka Streams application, which shuts down without any proper logging even at the debug level - 2020-12-18 14:25:36:875 +0000 [Thread-7] INFO o.apache.kafka.streams.KafkaStreams:? - stream-client…
Pratik
  • 1,122
  • 8
  • 26
0
votes
1 answer

SSL Setup between Filebeat and Amazon MSK (Kafka)

I have successfully got Filebeat exporting logs to MSK in plaintext mode. When trying to do the same over SSL I’m getting error messages from the Filebeat side Publish failed with circuit breaker is open and SSL Handshake errors in the MSK…
Garreth
  • 1,057
  • 2
  • 9
  • 24
0
votes
0 answers

How can I use an MSK trigger for Lambdas within a custom VPC?

I'm trying to use an MSK Lambda trigger with a Lambda that is connected to a VPC. However, when deploying the Lambda I'm always getting this error: Last processing result: PROBLEM: Connection error. Please check your event source connection…
Carlos Martinez
  • 4,350
  • 5
  • 32
  • 62
0
votes
1 answer

How change the "batch size" of a AWS Lambda MSK Event source

I created an MSK Cluster and a Lambda Function I want to be triggered when there are new events on the events on the MSK Cluster. I created the event source, and it says: Batch size: 100 I want a lower Batch size, but I cannot find any way to…
Nathan
  • 7,099
  • 14
  • 61
  • 125
0
votes
1 answer

Failing to connect to MSK cluster from ec2 host in the same vpc

I have a MSK cluster and I am not able to connect to it via an ec2 host in the same vpc. Here are my settings: Security group applied to EC2 (Outbound) Security group applied to MSK (Inbound) The ec2 instance profile role has AmazonMSKFullAccess…
prats
  • 225
  • 5
  • 15
1 2 3
…
24 25