I have setup 1 Zookeeper and 3 Kafka Broker (for Redundancy) on a single machine. I want to know what is the best practice for Kafka Setup on a single machine and multiple machines in a network. for e.g. if I set up on a single machine how many zookeeper, brokers and partition I should set up. Or If I set on multiple machines (N number of machines) then how many zookeeper, brokers and partition I should set up in respect to N.
Asked
Active
Viewed 306 times
1 Answers
0
As the machine is the main cause of failures, you gain zero by running thee copies on the same machine.
They will all fail at the same time.
Has QUIT--Anony-Mousse
- 76,138
- 12
- 138
- 194
-
what is the rules and best practices for partitioning topics...single partition is good or having 1,2,3 etc. partitions in multiple brokers? – Jason Foster May 09 '17 at 08:46
-
On a single host, you gain nothing by having multiple brokers etc. – Has QUIT--Anony-Mousse May 09 '17 at 09:19
-
I know; my question is about topic partitioning not multiple brokers on single host....as I know topic partitioning can help consumers work in parallel and boosting throughput. My question is how should we define number of partitions and what are the rules for doing so. – Jason Foster May 09 '17 at 12:08
-
As I investigated more it is suggested when we have N Consumers it would be better to have also N partitions to have parallelism. But having 2 partitions is not suggested because it may cause Consumer starvation and messages goes to only one partition leaving another partition empty....I am not sure why this occur when having 2 partition....I investigated more and found it somewhat related to Kafka Partitioner mechanism and underlying hashing mechanism that may have collisions. so I am not sure having 2 partitions instead of 1 or 3 is really bad thing!... – Jason Foster May 09 '17 at 14:15