Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
0
votes
0 answers

Terraform: Map existing iam_roles with corresponding iam_policies created dynamically

AWS MSK supports IAM based authentication, but not through resource based policy like S3. So we are trying to come up with Policy creation dynamically and attach it to an existing IAM roles of corresponding applications. Say there are two…
dilsingi
  • 2,938
  • 14
  • 24
0
votes
1 answer

Kafka-s3-connect killed instantly after start

I want to connect aws-Kafka with s3 using confluence connector on my ec2 server. I try to configure everything like in tutorials. When I run connect-standalone or connect-distributed, at first everything goes well, I don't get any errors in the logs…
Adam Gwóźdź
  • 1
  • 2
0
votes
2 answers

How to read the topics of a MSK cluster with React/TypeScript?

I have been trying to read this data (Topics) to my MSK Cluster from my React application, I have found very few tools and information that can help me, and I have not succeeded How can I get this data from MSK cluster and display them in a React…
Alison Quintero
  • 13
  • 3
0
votes
1 answer

json.Unmarshal and "push" to array of structs

I have a task to read records from AWS MSK (in AWS lambda) and prepare some kind of payload to send it to Facebook. Records from AWS MSK are base64 encoded, but once I decode them, I get JSON string. Now I don't understand how…
Valor_
  • 3,461
  • 9
  • 60
  • 109
0
votes
1 answer

Connection timeout when I try to connect to AWS MSK brokers

I created a cluster with 2 brokers in MSK but I cant connect using kafkajs client. I keep getting this connection timeout error {"level":"ERROR","timestamp":"2021-07-29T13:35:35.625Z","logger":"kafkajs","message":"[Connection] Connection…
Leutrim Osmani
  • 1
  • 1
  • 1
0
votes
0 answers

How to run Confluent Schema Registry using Docker on AWS ec2

I want to run schema registry for my AWS MSK cluster on EC2 within the same VPC as my MSK cluster using confluentinc/cp-schema-registry. But the container is exiting without any proper error message. Here is my docker command: docker run \ …
Venkat Papana
  • 4,757
  • 13
  • 52
  • 74
0
votes
0 answers

Java Producer for AWS MSK

As part of my existing application (which runs in the public network), I want to send some messages to AWS MSK Kafka cluster. As my Kafka cluster is within a VPC, I couldn't connect to it directly. That I understood. In my research, I came across a…
Venkat Papana
  • 4,757
  • 13
  • 52
  • 74
0
votes
0 answers

Unable to connect to MSK Kafka Broker after adding advertised.listener config

I have recently been adding advertised.listeners property to my MSK cluster configuration. The command used was /usr/local/kafka/bin/kafka-configs.sh --bootstrap-server b-1.xxxxxxxxx.c4.kafka.xxxxxxxxx.amazonaws.com:9094 \ --entity-type brokers…
floating_hammer
  • 409
  • 3
  • 10
0
votes
1 answer

How can we update or create new MSK trigger for a lambda function via api

I am trying to update the existing MSK trigger my lambda function by adding topics to the list via API. If Editing is not possible I am also open to creating a new MSK trigger. This new trigger will be added to the existing lambda via another lambda…
Tannu Priya
  • 313
  • 2
  • 15
0
votes
1 answer

AWS MSK (Kafka) LEADER_NOT_AVAIABLE while pushing event to topic

I've provisioned MSK, 2 nodes, while pushing mesage to topic get success but with following warning 18:33:06.318 [kafka-producer-network-thread | producer-1] WARN o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error…
Andriy Kopachevskyy
  • 7,276
  • 10
  • 47
  • 56
0
votes
1 answer

AWS MSK vs Confluent for hosting Kafka?

In terms of investing for the most value, how does AWS MSK compare to Confluent when it comes to hosting an end to end Kafka event sourcing? The main criteria to be used for comparing are: Monitoring Ease of deployment and configuring Security
Amit Dash
  • 584
  • 8
  • 21
0
votes
1 answer

Unable to send data from fluentd to Amazon Hosted Kafka

I created an AWS MSK cluster, that use SASL/SCRAM Authentication as mentioned in the docs. I am trying to send logs from fluentd to this AWS MSK using the fluentd-plugin-kafka Below is the output plugin configuration I used @type…
Bhavya Latha Bandaru
  • 1,078
  • 12
  • 21
0
votes
0 answers

AWS MSK working with shell command but not with python

I created a AWS MSK instance with two brokers. I can use that to produce and consume messages by using kafka#s shell commands (kafka-console-producer.sh and kafka-console-consumer.sh) but the problem is, it is not working with kafka python…
Mainul
  • 13
  • 5
0
votes
2 answers

Unable to use IAM Access control method using terraform aws_msk_cluster resource

I am trying to develop a module to create AWS MSK. I would like to enable IAM authentication for MSK resource I am following the below link, but I don't see anything related to IAM…
Ram
  • 655
  • 2
  • 7
  • 27
0
votes
1 answer

AWS MSK - Update storage volume size by modifying the CF

I have created an MSK cluster and all associated configuration using cloud formation template. Now I have to update the stack with more storage. Can I just modify the CF and specify the new storage requirement in the template and run the CF again? …
dvlpr
  • 311
  • 3
  • 17
1 2 3
…
24 25