1

I have a Kafka topic on my spring boot application on which I am sending some data from producer. I want to check the load of my topic so that I can create new topics, if load on the previous topic exceeds

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Fardeen mirza
  • 75
  • 1
  • 6
  • 1
    That sounds like a wrong usage of Kafka. Anyway why creating new topics instead of leveraging Kafka horizontal scale by adding more partitions? – Noam Levy Jan 11 '22 at 10:05
  • It was a requirement for adding more topic for different users, but now I am managing it with partitions. Thank you for your response. – Fardeen mirza Jan 13 '22 at 05:39

1 Answers1

0

Topics don't have "load" in the traditional sense. Sure, you can use JMX metrics to measure incoming byte rates, but that is network load, as measured by the broker. You can also measure outgoing rates by the producer, per partition, and aggregate to get data by topic.

The brokers hosting the partitions do have load; measurable network, disk , and CPU load.

Secondly, your producers would all need to be updated to actually send data to those new topics you'd created; they nor the brokers would know to "distribute load" to them.

The correct way to reduce broker load, and distribute data to more brokers is to increase the cluster size, and the correct way to scale production is to add more partitions.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245