Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
1
vote
1 answer

Meaning of SumOffsetLag in MsK cloudwatch data?

We have kafka which we are monitoring using grafana dashboad provided by aws cloud watch. From the past few days we are seeing some abnormality. I dont have Idea much about SumOffsetLag metric parameter. How this spiked value can degrade the…
dead programmer
  • 4,223
  • 9
  • 46
  • 77
1
vote
1 answer

AWS MSK Spring Boot App Example Using IAM

I want to create a Spring Boot App that connect to AWS MSK using IAM and produces and consumes event i looked over the internet but was not able to find any good resources Note: I have deployed my a demo app hello world app on AWS EKS.
giiyiraj
  • 51
  • 1
  • 6
1
vote
0 answers

Timed out waiting for a node assignment. while creating topic in provided MSK cluster

I have EC2 as a client to create topics and when i execute the command to create it, it fails, I'm using this documentation https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html but is not working for me.
Douglas Guzman
  • 29
  • 5
1
vote
0 answers

Not able to publish messages to Kafka from Lambda function

I am trying to publish some messages to kafka from a aws lambda function. When I'm trying to test this feature on my local, the messages are not getting published and the function is getting timed out. I was able to connect to the local kafka…
pkgajulapalli
  • 1,066
  • 3
  • 20
  • 44
1
vote
1 answer

How to use AWS_MSK_IAM sasl mechanism with a kafka producer jar?

I have a fat jar called producer which produces messages.I want to use it to produce messages to a MSK serverless cluster. The jar takes the following arguments- -topic --num-records --record-size --throughput --producer.config…
Gestalt
  • 93
  • 10
1
vote
0 answers

Why does Kafka partitions get Under Replicated?

Our production Kafka cluster runs on 3 nodes of kafka.m5.large type on AWS MSK. Some of our topics have 36 partitions with a replication factor of 3. We often notice that due to some unknown reason, the count of Under Replicated Partitions grow up…
aveek
  • 188
  • 6
1
vote
1 answer

Get latest revision of AWS::MSK::Configuration in CloudFormation

I'm trying to create a cloudFormation stack with MSK Configuration and associating MSK Configuration with MSK Cluster. Creation of AWS::MSK::Configuration returns only ARN while I need ARN and Revision number to associate MSK Configuration to MSK…
webh
  • 329
  • 4
  • 6
1
vote
0 answers

Consumer connect errors from AWS MSK via IAM - SSL problem or something else?

"thread_name":"org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1","message":"[Consumer clientId=consumer-groupId-1-1, groupId=groupId-1] Initiating connection to node node.kafka.us-west-2.amazonaws.com:9098 (id: -2 rack: null) using…
acme-j
  • 99
  • 10
1
vote
0 answers

AWS MSK posting to topics very slow

We have the same architecture under two different accounts on AWS. One is our development environment, and the other is our QC environment. In our QC environment, it takes (suspiciously) 60 seconds, give or take a millisecond or two, to post to a…
SteveB
  • 483
  • 1
  • 4
  • 18
1
vote
0 answers

AWS CDK MSK get bootstrap server list for an existing cluster

I am trying to deploy AWS Fargate services, written in Spring boot to consume messages from an existing MSK Kafka cluster using AWS CDK. I can get the ICluster reference using the method const kafkaCluster = msk.Cluster.fromClusterArn(...). But…
Manu
  • 3,467
  • 3
  • 29
  • 28
1
vote
0 answers

MSK connector to handshake with a third party hosted cluster using Kafka SSL or SASL Authentication

I have created a kinesis source connector which is supposed to pull data from the kinesis stream and push the same data to kafka topic. The use case here is connector will be deployed on Amazon MSK connect and where as the topic is created on some…
Yogesh Katkar
  • 133
  • 10
1
vote
0 answers

AWS MSK Public cluster authentication with SASL/SCRAM

The configuration of AWS MSK is public and a secret is attached to it.The security setting is set to SASL_SCRAM enabled, allow.everyone.if.no.acl.found=false and auto.create.topics.enable=true is set as written. I'm trying to connect to the cluster…
gundocan
  • 11
  • 2
1
vote
1 answer

Unable to connect with MSK via Couchbase-Kafka connector

We are trying to connect with a IAM Auth enabled MSK via Couchbase Kafka connector. Below properties are added to $KAFKA_HOME/config/connect-standalone.properties in addition to regular configs…
Bond - Java Bond
  • 3,972
  • 6
  • 36
  • 59
1
vote
0 answers

How to get the latest offset of a Kafka Topic through AWS Lambda function

I have created a AWS Lambda function which should connect to given Kafka Topic and return the latest offset of it. But it is not working. exports.handler = async (event, context) => { var Client = kafka.KafkaClient; var client = new Client({…
Ayan rahate
  • 21
  • 4
1
vote
1 answer

AWS SAM template with multiple MSK topics

I have a Lambda function which tiggers off 3 MSK topics. I have found I can easily define this dependency in my template.yaml, but only for 1 topic: Resources: MyFunction: Type: AWS::Serverless::Function Properties: CodeUri: src/ …
Jeppe
  • 1,424
  • 2
  • 15
  • 36
1 2 3
…
24 25