0

I am using Kafka 2.9.2-0.8.1 version..

From the documentation, it seems that load balancing is done automatically for the configured cluster.

Here is my Java producer configuration:

Properties props = new Properties();
props.put("batch.size", "200");
props.put("producer.type", "async");
props.put("connect.timeout.ms", "5000");
props.put("request.required.acks", "0");
props.put("metadata.broker.list", "10.10.73.52:9092,10.10.70.15:9092");
props.put("serializer.class", "kafka.serializer.DefaultEncoder");
props.put("partitioner.class", "kafka.producer.DefaultPartitioner");

Note: I have kept all the default configurations, which were provided by the Kafka distribution..

Zookeeper seems to have discovered my other broker : 10.10.70.15.. when I checked the logs..

I have created a test-topic on one of the brokers.. using the console-producer.sh.. which then created appropriate directory in /tmp/kafka-logs folder in all the other registered brokers in zookeeper.

--> ./kafka-topics.sh --create --zookeeper 10.10.73.52:2181 --replication-factor 2 --partitions 2 --topic test-topic

I have used the following provided line to subscribe to the topic on both broker machines..

--> ./kafka-console-consumer.sh --zookeeper 10.10.73.52:2181 --topic test-topic

Producer code:

KeyedMessage<String, byte[]> publishData = new KeyedMessage<String, byte[]>("test-topic", data);
producer.send(publishData);

I see that both the brokers receive the same data.. and load is not balanced.

Do I need to implement any other load-balance/partition logic?

Any ideas what am I missing here?

First Blood
  • 235
  • 1
  • 7
  • 21
  • 2
    It looks like you're using replication factor 2. So all values are replicated to 2 machine. To see load balancing in action either set replication factor to 1 or add third broker. – Wildfire Apr 15 '14 at 10:34
  • I see..Thanks for the reply.. So the replication factor should not be equal to the number of brokers? So, adding 3 brokers, with a replication factor 1, should load-balance? Also, I believe I was checking for load-balancing thingy incorrectly.. I started the console-consumer on both my brokers, listening for same topic.. probably that's why I was seeing the same data? How does that work? – First Blood Apr 15 '14 at 19:42

0 Answers0