2

We have a sandbox that has 5 nodes with all five nodes running a kafka broker(broker id = 0) Now I have made copied of config files on all 5 nodes with distinct broker id's and log files directory to have multiple broker running

-rw-r--r-- 1 root root 5652 Apr  2 23:01 server.properties - (this one being the default)
-rw-r--r-- 1 root root 5675 Apr  2 23:02 server1.properties
-rw-r--r-- 1 root root 5675 Apr  2 23:02 server2.properties

Now do I start kafka on all the 5 nodes with new config files by using 

./kafka-server-start.sh config/server1.properties &
./kafka-server-start.sh config/server2.properties &

Does every node will have 3 three brokers running?? or its 3 brokers for overall cluster?? how does this work?Any help would be appreciated??

Suren Baskaran
  • 1,228
  • 1
  • 10
  • 17

1 Answers1

2

Each node in your cluster should only have one configuration file and kafka-server-start should only be run once on each node. For example, server 1 only needs to have a single configuration file that contains, e.g. broker.id = 1.

Each time you run kafka-server-start you are starting a broker (i.e. server). When the broker starts, Kafka will locate the other brokers via ZooKeeper. This allows new brokers to be added and removed from the cluster without any additional configuration file specifying the other nodes in the cluster and without having to do any reconfiguration on the existing nodes.

If you're running kafka-server-start multiple times on the same node then you are indeed starting multiple brokers on the same node, but that's not what you want.

kuujo
  • 7,785
  • 1
  • 26
  • 21
  • Thnx for the response - but my intented design was to to have 3 brokers running on each cluster and Yes I ran kafka server start 3 times for each config files, so I would have 3 broker per node – Suren Baskaran Apr 03 '15 at 21:47