Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
3
votes
0 answers

Has anyone figured out how to use KSQLDB with AWS GLUE Schema Registry?

We're trying to use AWS MSK (managed Kafka), and we want to use AWS GLUE Schema registry with AVRO rather than Confluent Schema Registry. We have brought up KSQLDB, and connected that to MSK, but it needs ksql.schema.registry.url and all that is…
ChrisAddams
  • 190
  • 1
  • 8
3
votes
1 answer

Confluent S3 Sink Connector for Kafka unable to connect to S3 using MSK / MSK Connect

I'm attempting to store my Kafka messages to S3 using Apache MSK / MSK Connect and the Confluent S3 Sink Connector. I'm experiencing a timeout error when the connector tries to reach S3. To rule out a permissions issue, I've given the connector role…
NRaf
  • 7,407
  • 13
  • 52
  • 91
3
votes
2 answers

AWS MSK trigger for AWS Lamba - multiple topics in the same execution context

I have a Lambda function that has multiple MSK trigger configurations - each one for different topic. Didn't find any information in official documentation if Lambda's input (MSKEvent) can contain multiple different topics. Official docs provides…
zooblin
  • 2,172
  • 2
  • 27
  • 33
3
votes
0 answers

Changing the number of partitions in AWS MSK with no downtime

Still fairly new to kafka. Trying to change the number of partitions without any downtime/data loss. Currently have 1 partition with 2 brokers and I want to move to, say 10 partitions with 3 brokers (happy to have 10 partitions on each broker).…
friartuck
  • 2,954
  • 4
  • 33
  • 67
3
votes
1 answer

MSK vs SQS + SNS

I am deciding if I should use MSK (managed kafka from AWS) or a combination of SQS + SNS to achieve a pub sub model? Background Currently, we have a micro service architecture but we don't use any messaging service and only use REST apis (dont ask…
Rachit
  • 208
  • 4
  • 19
3
votes
2 answers

Define Kafka ACL to limit topic creation

We are currently running an unsecured Kafka setup on AWS MSK (so I don't have access to most config files directly and need to use the kafka-cli) and are looking into ways to add protection. Setting up TLS & SASL is easy, though as our Kafka cluster…
Dennis van de Hoef - Xiotin
  • 1,481
  • 1
  • 13
  • 29
3
votes
0 answers

MSK bootstrap server endpoint fixed between cluster recreations

What would be the recommended way to achieve static ip addres or DNS A record or alias for brokers that does not change on MSK cluster recreation? Considering how rare this is supposed to happen, putting an NLB in front of it seems like an overkill…
hdjur_jcv
  • 686
  • 1
  • 12
  • 30
3
votes
1 answer

Connect to MongoDB atlas within AWS VPC using AWS MSK connector

I am trying to use MongoDB change stream using Kafka. I chose AWS MSK since my entire infrastructure is within AWS and it will be easy to integrate with other AWS services. I created an AWS MSK cluster within the VPC and created an AWS MSK connect…
Jerin A Mathews
  • 8,572
  • 4
  • 26
  • 49
3
votes
0 answers

Can you use `kafkacat` with AWS IAM Access Control

I'm trying to use IAM Access Policy for authorization with MSK. I was trying to follow https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#configure-clients-for-iam-access-control. Looks like kafkacat doesn't support it.…
idetyp
  • 486
  • 1
  • 3
  • 13
3
votes
1 answer

AWS DMS Kafka target endpoint testing failed showing "application error 1020912: failed to connect to database"

I created a DMS instance in a public subnet without public access. Kafka is created as an MSK cluster in a private subnet, within the same VPC. No SSL. Endpoint testing shows application error 1020912: failed to connect to database. I can confirm…
dz902
  • 4,782
  • 38
  • 41
3
votes
0 answers

MSK Kafka connect commit of offsets timed out

I am using Kafka connect to take messages from an MSK cluster, and put them on a Kinesis firehose as per https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-kafka-connector-msk/ I have the setup working, but when I try to put large…
thewire247
  • 795
  • 1
  • 9
  • 24
3
votes
2 answers

How to connect from spring boot kafka project to aws MSK

I am trying to connect to a aws msk cluster from spring boot kafka app. i have aws msk arn ,boot strap server config available. How to integrate in the application? Any examples Can i run it from local,i mean connect to AWS MSK from local app …
sc cc
  • 75
  • 2
  • 7
3
votes
3 answers

AWS MSK User/Password Authentication/Authorization

i want to be able to authenticate/authorize clients to produce/consume messages on certain topics. they would be part of our vpn (incl. aws). as i understand the available documentation the only option to do this is to issue client certificates and…
niesfisch
  • 93
  • 1
  • 6
2
votes
1 answer

How to approach configuration as code for Kafka topics in AWS MSK?

Does AWS MSK support creating topics via configuration, similar what Strimzi does?
clearpath
  • 916
  • 8
  • 24
2
votes
0 answers

fluentd kafka frequent connection closes - Kafka High cpu

We writing from around 1000 ec2 machines to AWS MSK using kafka fluentd plugin. They write continuously throughout the day. The aws.kafka.connection_creation_rate metric graph and the aws.kafka.connection_close_rate metric graph are pretty high…
Gaurav Shah
  • 5,223
  • 7
  • 43
  • 71
1 2
3
…
24 25