Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
0
votes
1 answer

How to load test an asynchronous pipeline?

I have the following pipeline in AWS API-Gateway -> Lambda -> Kafka(Amazon MSK) -> Consumers I need to load test the entire pipeline and each component in the pipeline (to identify bottlenecks) I don't have any prior experience in load testing. So,…
sai
  • 59
  • 4
0
votes
1 answer

MSK has no security, when creating a Rule Action from IoT Core to Kafka it wants me to select security, can I avoid this?

As the question states, when the MSK was set up - it had no security enabled, and in turn it's causing issues creating the rule itself. When creating the rule through the AWS GUI, the action username and password property must be configured using…
veila
  • 83
  • 8
0
votes
0 answers

Create MSK Connector from AWS Lambda function

I'm trying to create an MSK Connector from a Lambda function. Code right now is something as simple as this: import json import boto3 client = boto3.client('kafkaconnect') def lambda_handler(event, context): response =…
André Machado
  • 726
  • 6
  • 21
0
votes
0 answers

AWS MSK - Brokers going out of service all of a sudden

We had setup an MSK cluster with 3 brokers (each have 2 GB storage). The cluster was working fine for first day, then all of a sudden it stopped consuming messages on 2nd day even though there was no config change. The error we were getting at…
VRASHAB
  • 1
  • 1
0
votes
0 answers

AWS MSK - Internal Brokers communication

I am using AWS MSK for our production workload and we have been noticing some not very clear log messages in cloudwatch. The messages are about the internal communication between brokers (more on cluster setup later): [2022-05-14 06:50:17,171] INFO…
AlessioG
  • 576
  • 5
  • 13
  • 32
0
votes
0 answers

MSK & building aggregrate tables (e.g. for analytics)

I use MSK and I manually build aggregate tables of my streams in my application code (e.g. TypeScript in a node.js webservice). I have lots of data (approaching 1M events per day), and I want to be able to productionise different real-time 'views'…
friartuck
  • 2,954
  • 4
  • 33
  • 67
0
votes
1 answer

AWS MSK. Kafka Connect - plugin class loading

I am using Kafka Connect in MSK. I have defined a plugin that points to a zip file in s3 - this works fine. I have implemented SMT and uploaded the SMT jar into the same bucket and folder as the zip file of the plugin. I define a new connector and…
balderman
  • 22,927
  • 7
  • 34
  • 52
0
votes
1 answer

AWS MSK Connect with Mongo DB as the Source

We are planning to have an MSK created in AWS with Mongo DB as the Source for the MSK. We do not see any Pre-defined Connectors for connecting to the Mongo DB as source. How to setup a connector? Do we need to create a Connector completely or are…
Silly John
  • 1,584
  • 2
  • 16
  • 34
0
votes
1 answer

Connection error when set MSK as trigger of AWS Lambda

I'm trying to set MSK as a trigger of AWS Lambda. But there is a problem like the one below. [PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide…
LYG
  • 41
  • 1
  • 7
0
votes
2 answers

Run kafka consumer on EC2

I made MSK cluster, and sent a message from my Producer. It works well. And, I want to make my Consumer get the message. The Consumer in a python file, and the python file is in EC2. How can I run the python file and make my Consumer get the…
LYG
  • 41
  • 1
  • 7
0
votes
0 answers

Multiple small kafka clusters or big shared cluster using AWS MSK

We are trying to setup Kafka clusters using AWS MSK (Managed streaming for Kafka). We have multiple use cases where we may need >50 topics each having 5-25 partitions and cumulatively ~100 million events being produced in a single day. Producer…
kushan85
  • 102
  • 1
  • 9
0
votes
1 answer

AWS MSK Can not Connect from Azure

I have set up in Aws inside 1 VPC and configured AWS MSK(Kafka) across the AWS VPC I am able to connect the MSK using bootstrap URL. But When I try to connect from Azure VM I am not able to connect. I have created a site-to-site VPN connection…
ankitbeohar90
  • 109
  • 13
0
votes
2 answers

AWS MSK Connectors for Debezium IAM Role Problem

I want creat a connector for Debezium. I am unable to create a kafka connector using the AWSServiceRoleForKafkaConnect role. Here is the error I get when clicking Create Connector on the last page in the form: Error creating connector There was a…
ahmetucan
  • 1
  • 1
0
votes
1 answer

How to create MSK Cluster configuration with Cloud Formation?

I'm creating MSK with Cloud Formation. By the way, I can't find how to create MSK Cluster configuration, in Cloud Formation documents. So how can I create MSK Cluster configuration with Cloud Formation or do I can't create MSK Cluster configuration…
정원희
  • 13
  • 2
0
votes
1 answer

How to find out if SSL/TLS is mandatory on a listener port of kafka broker?

This might be done by issuing openssl s_client -connect with -no_tls1_2 -no_tls1_1 -no_tls1 -no_ssl3 -no_ssl2 but the result seems inconclusive to me. The listener is configured with SASL/SCRAM authentication enabled, but I am not sure if Encrypt…
hdjur_jcv
  • 686
  • 1
  • 12
  • 30
1 2 3
…
24 25