11

I had a question regarding Kafka broker clusters on AWS. Right now there is an AWS ELB sitting in front of the cluster, but when I set the "bootstrap.servers" property of my producer or consumer to the "A" record (and correct port number) of my ELB, both the producer and consumer fail to produce and consume messages respectively. I have turned off all SSL on my broker and am connecting through the PLAINTEXT 9092 port, with which my ELB forwards port 1234 to 9092. So in my Producer Configs properties for example, I will have...

bootstrap.servers = ("A" record of ELB):1234

More info:

  • My ELB's protocol is TCP/TCP
  • My "advertised.listeners" property on brokers are PLAINTEXT://(ec2-private-ip):9092

Has anyone had any luck running Kafka behind an ELB? If so, please help me out!

Mattnv92
  • 419
  • 1
  • 4
  • 9

2 Answers2

12

You can use an ELB as the bootstrap.servers, but the brokers still need to be directly accessible to the client. The ELB will be used for the initial metadata request the client makes to figure out which topic partitions are on which brokers, but after that it'll use the hostname of the server (or advertised.listeners setting if you need to customize it, which, e.g. might be necessary on EC2 instances to get the public IP of a server).

Ewen Cheslack-Postava
  • 1,411
  • 11
  • 11
  • I'm still a little confused. Doesn't the ELB talk to its instances via private IP, so wouldn't I want the advertised.listeners to be the private IP? I've tried both the public and private IPs of the ec2s and it's not working for either. I am using kafka client library for Java by the way. – Mattnv92 Aug 01 '16 at 18:18
  • The EC2 instance running Kafka could listen on a single interface but be able to accept traffic targeted at both the public and private IPs. What I was trying to get at is that the ELB is *only* used for the initial request to get metadata about the cluster. After that, the client needs to be able to talk *directly* with the leader broker for each topic partition. If you were trying to use an ELB to make a Kafka cluster publicly available, you'd need to make sure the `advertised.listeners` for each broker also makes it publicly accessible. – Ewen Cheslack-Postava Aug 01 '16 at 22:16
  • 2
    I feel so dumb. So it was actually working, but the ELB was reporting that the instances were "out of service" because the health check failed. It was pointing to port 80 by default on our broker which isn't "pingable". So the ELB wasn't ever routing the traffic to any of the brokers. We have ELB now pinging the Zookeeper port on our broker and it is working just fine! Knew it had to be something silly :p. Thanks for the help! It definitely shed some light on how Kafka works, and I'm sure will help others in the future! – Mattnv92 Aug 02 '16 at 19:56
0

IMHO, It is not suggested to add additional SLB to kafka cluster.

  • Kafka cluster are decentralized in essence, SLB add single point of failure risk and complicate the cluster setup.
  • Message send and receive are network lag sensitive, SLB will reduce the message throughput definitely.
Shawn Guo
  • 3,169
  • 3
  • 21
  • 28
  • I would have expected more than opinion here. The person asked a good question and -- while I almost share your opinion -- I didn't read anything in it it that answered the original question. – staylorx Mar 28 '18 at 15:35