2

I have 3 node kafka cluster (zookeeper also installed on the same 3 nodes). Im not sure if I deploy the AWS NLB in front of my broker. I have 3 producers, even though if the evenly go to all 3 brokers, they will decide where to partition it and etc. I don't know what benefit I can get from AWS NLB and what is the cons of it.

TheDataGuy
  • 2,712
  • 6
  • 37
  • 89

1 Answers1

1

I also researched this and didn't find much help out there. I ended up putting a NLB, with a TCP target group, in front of my broker and this is why:

  1. Save some DNS headache. I have a CNAME on the NLB A record and that's what I use for my bootstrap server value. I can scale horizontally seamlessly by just adding the new broker to the NLB target group (via Cloudformation). I'm not tied down to any IPs in our AWS environment now because of DNS records. I also use a Route53 private zone for the Zookeeper nodes so the brokers only point to the overall A record that all those nodes share.
  2. Easy to monitor broker health with built in CW monitoring.
  3. I read about the benefit of SSL offloading with an ELB, but I don't really consider that a benefit because the client to broker comms will still be non-SSL. I'm not doing that, but I thought I'd list it.

I haven't done any benchmark testing with the NLB yet, but I'm not too concerned. IMO, the simplified DNS made it worth it.

Cheers

EDIT: Proxy protocol will not work with Kafka, so if you want the ability to restrict traffic via source IP, in your security groups, you'll have to use type 'instance' vs type 'ip' for your NLB target group targets.

https://aws.amazon.com/premiumsupport/knowledge-center/security-group-load-balancer/

A lesson learned using the NLB name from a target:

https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/

For this issue, I just switched my --bootstrap-server to 'localhost' on any broker target.

Urton
  • 36
  • 4