3

I was asked a question to achieve scaling a system. The system itself processes customer data and creates filtering on that data and produces some analytical information.

1) The First iteration:

My initial answer was to provide a Kafka cluster solution.

Kafka itself has streaming , load balancing, and fault tolerance capacity. So it provides me to create processing the producing data in many brokers effectively and consume this data in any consumer as I want.

I can also add streaming functionality to filtering data as much as I want.

In this scenario, there is no need to think about the load balancers, because the Kafka handles the load balancing itself.

2) The second iteration: It has been asked to me how do I scale the system if there is huge demand increase. What would be the approach to scale the system?

In this case, when considering the Kafka cluster; broker counts and partitions should be described in the beginning. It is not something expanding itself. So even though Kafka provides lots of flexibility when considering multiple locations and fastly increasing requests my second opinion is using Elastic Load Balancers & automatic binding for data centers.

When the request counts double in the next day. Load balancers route the load other load balancers/ other data centers so in case of necessity the new Kafka clusters automatically connect to the total system.

The main load routing could be done geographically.

The point of this issue load balancers is still looks needed although Kafka is so strong candidate.

My second approach similar to the following schema.

https://i.stack.imgur.com/kEx1C.jpg

(In the meantime, I came across some interviewers before than this interview and they exactly named load balancer as an "out of date technology" and I have been judged very cruel because I suggested load balancers. )

If you are a Kafka expert & and dealing with scaling of increasing requests in multi-location, I would be glad if you put your comments.

Thanks.

Neslihan Bozer
  • 179
  • 3
  • 12

3 Answers3

1

While you can place TCP load balancers in front of Kafka brokers, it only creates another point of failure, and IMO is pointless because clients must send requests to the partition leaders directly, which a load-balancer would not have context of unless configured to do so.

From the Kafka Protocol docs

the client needs to somehow find one broker and that broker will tell the client about all the other brokers that exist and what partitions they host. This first broker may itself go down so the best practice for a client implementation is to take a list of two or three URLs to bootstrap from. The user can then choose to use a load balancer or just statically configure two or three of their Kafka hosts in the clients

HAProxy or Nginx are not "outdated". You'd need more clarity from the person that said this

If there is a message production increase, then the Kafka consumer settings could be adjusted to handle back-pressure. Only when the brokers are approaching hardware limits, should more resources be added (not only during spiky load)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • "HAProxy or Nginx are not "outdated", although "service mesh" & "API Gateway" might be the "new" terms used to describe other features they provide." Can you be more descriptive about this phrase? – Neslihan Bozer Feb 06 '20 at 12:42
  • "If there is a message production increase, then the Kafka consumer settings could be adjusted to handle back-pressure. Only when the brokers are approaching hardware limits, should more resources be added (not only during spiky load)" . How could we manage the high increase in multi-location. How do we decide automatically to increase resource in which location? – Neslihan Bozer Feb 06 '20 at 12:44
  • 1) You said your interviewers were cruel when you mentioned load balancers. Well, first, if interviewers are cruel, you don't want to work with those people. Second, it's not clear what they expected instead. Because those tools I mentioned are popular software load balancers still used today – OneCricketeer Feb 06 '20 at 13:47
  • 2) I'm not sure about "automatic", but it requires collecting metrics and logs within your application to know if it's not able to process events as quickly as you're expecting. You'll have to change the code and redeploy the apps – OneCricketeer Feb 06 '20 at 13:50
  • maybe I don't wanna work specifically with that person but it is not the point, those kinds of people cost you a job in a good company. The answer was he expecting was that I should suggest using only Kafka. – Neslihan Bozer Feb 06 '20 at 14:03
  • Thanks for the valuable comments but I still didn't get the answer to the exact question. The original question "Does Kafka replace the Load Balancers completely? ".And also If HAProxy and Nginx are still valid approaches should we consider using it only without Kafka usage? – Neslihan Bozer Feb 06 '20 at 14:09
  • 2
    No, Kafka cannot load balance HTTP traffic. Period. Kafka has its own protocol. – OneCricketeer Feb 06 '20 at 14:17
  • service mesh is not even similar to API gateway, and neither of those terms are new terms to describe any features of haproxy or nginx, which are both software packages. The entire phrase is just wrong on so many levels. – Software Engineer Apr 15 '21 at 18:07
1

Load-balancing with Kafka would run into problems, as the client itself is going to create multiple connections to Kafka brokers, possibly bypassing your proxies. On startup, clients (producers/consumers) send Metadata requests to the bootstrap.servers figure out how the cluster looks like. This can be observed in detail when you turn on trace/debug log levels in your Java clients.

The mesh-level solution, on the other hand, is going to need protocol support, e.g. something like this is happening in Envoy - https://github.com/envoyproxy/envoy/issues/2852

Adam Kotwasinski
  • 4,377
  • 3
  • 17
  • 40
  • So do you think should we avoid load balancers totally while using Kafka clusters? – Neslihan Bozer Feb 06 '20 at 12:46
  • "The mesh-level solution, on the other hand, is going to need protocol support, e.g. something like this is happening in Envoy - https://github.com/envoyproxy/envoy/issues/2852" could you describe this in more detail please? – Neslihan Bozer Feb 06 '20 at 12:47
0

if your client is an internal one, you can get away without using any interface that provides support for external client (such as public ones over HTTP), so you dont need to have any ALB or API Gateways. But if you need to support the external clients, using a load balancer for routing or directing the heavy traffic is perhaps the wisest strategy. You may hand the load directly to a lambda which takes care of processing, or you may dump it into Kafka for a "fanout" (multiple consumers working on the messages). It would depend on the load, use cases and replayability.

Amar
  • 1