0

I have created a cluster in AWS MSK and created a topic. I can produce and consume events from the EC2 instance in the same VPC as the cluster. The specific use case is that I want to consume events using Java. The KafkaClient AWS SDK doesn’t have any methods for event consumption, so I can’t use it here. I cannot directly pass the bootstrap servers in the Properties() while creating a KafkaConsumer as the IPs are internal to the VPC.

I have gone through this AWS guide on ways to connect to MSK but not sure which method would be suitable for this exact use case. I have tried out a few ways-

  1. Setting advertised.listerners to the public IP of the instance. But still kept getting this error while listing topics after creating KafkaConsumer- Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

  2. Opening relevant ports (9092 for PLAINTEXT in my case) in the security group inbound rules. I was still getting a Connection refused error while trying to telnet to port 9092 after doing this.

My question is will any of the above work and I'm just missing something else here which can solve this. Or will the above methods won’t work?

My end goal here is to consume events from MSK Cluster using Java.

sd99
  • 13
  • 4
  • Is your java process going to run in the same VPC or outside? If outside, do you expect your MSK cluster to be available publicly to all ? – Rishabh Sharma Mar 11 '21 at 18:54
  • @RishabhSharma It needs to run from outside (i.e. not from the same VPC). Yes, I want it to be available publicity (i.e. it should be accessed from anywhere if the correct client.properties are passed). For this I guess we need to make the IPs of bootstrap servers publicly accessible, since they are currently private to the VPC. – sd99 Mar 14 '21 at 06:56

1 Answers1

1

Since you need to make the MSK Endpoints ips public, you need to improve your authentication process and place the MSK cluster on a public subnet in a VPC that have a Internet Gateway and properly configured route tables: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html

If you connect this, even being public, in your own VPC, you can use a CLI to retrieve the bootstrap servers from MSK Cluster and inject this as an environment variable using user data in EC2 bootstraping: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html

A command to retrieve bootstrap servers from MSK Cluster using CLI:

aws kafka get-bootstrap-brokers --region us-east-1 --cluster-arn ClusterArn

More documentation about MSK cluster configuring: https://docs.aws.amazon.com/msk/latest/developerguide/produce-consume.html

Igor Eulálio
  • 91
  • 1
  • 7