0

We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )

Since we are dealing with Kafka production cluster , we want to understand what is the best Practice and the right approach to install the machines and configuration

The question about the cluster that we are preparing is about the controllers

Do we need to set and install the controllers on dedicated machines and not on Kafka machines itself?

for example I gives here PLAN1 and PLAN2

PLAN1

We can install 3 controllers on the first 3 Kafka machines ( means both PID's are running on Kafka machine as broker pid and controller pid [ pid - Linux process id ]

kafka1-3 will include pid for broker and pid for controller

kafka4+ will include only pid for broker

enter image description here

PLAN2

We can install 3 controllers on different dedicated machines

35 physical machines for Kafka Kraft service

3 dedicated VM machines for Kafka controllers ( VM machines since from our understanding controllers are not consume so much resources )

enter image description here

The second question , is about how many controllers services we need?

Is it always 3 controllers for 3 Kafka machines or for 15 Kafka machines or for 35 Kafka machines ?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
jessica
  • 2,426
  • 24
  • 66
  • 1
    This looks more like a question on deployment strategy rather than _programming_, so is outside the scope of Stack Overflow. Perhaps see if it's appropriate over on [swe.se]? – Toby Speight Sep 01 '23 at 10:09

1 Answers1

1

Think about it like this - Raft is a leader election protocol. If you lost 1/3 of the nodes; you're left with 2 equal "electors" who may not agree on a decision.

So, having more controllers is better for quorum, but you generally don't need more than 5 or 7 (odd numbers recommended). This didn't really change from Zookeeper or Kraft...

Two "pid" on any node isn't what you want. process.roles accepts both roles of broker and controller, and would run as a single Java process. You can also separate them, and ultimately is determined by if you have the hardware available and how many nodes you're willing to lose

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • about what you wrote - "Two "pid" on any node isn't what you want" you means that kfka broker machine will have one PID that include both - broker service and controller ?? – jessica Sep 01 '23 at 13:26
  • 1
    Yes. Test it locally by starting Kafka with that setting – OneCricketeer Sep 01 '23 at 13:27
  • in any case form your ( great ) answer we understand that preferred solution i to set the controllers on different machines as my PLAN2 , am I right ? – jessica Sep 01 '23 at 13:28
  • 1
    Separate machines would be recommended yes, but they don't need to be identical hardware as the brokers, per my understanding. – OneCricketeer Sep 01 '23 at 17:23