1

I'm new to kafka and preparing use it for production.

What strategies can be used for rebalancing data storage if brokers for a topic's current partitions are running out of disk space, if more brokers can be added to the cluster?

By a simple example, say a topic has 3 partitions at beginning (1 replica to simplify problem), and 3 brokers each stores 1 partition of the topic, and each of these partition takes up 1TB disk space.

How can I add 3 more new broker servers and alter topic's partition amount to 6, and end up with a data rebalance result of each of the 6 partitions takes up 500GB disk space on its broker?

I think this problem is critical for storing large amount of data forever in kafka cluster.

Thanks.

Zachary
  • 11
  • 1

1 Answers1

0

You can rebalance the cluster https://www.linkedin.com/pulse/partitions-rebalance-kafka-raghunandan-gupta, or if you have kafka manager(https://github.com/yahoo/kafka-manager) you can do it from there.

Alex H
  • 1,814
  • 11
  • 18