3

Can I use kafka as the messaging queue in system which recieves end user events as messages there will be short bursts of load (lasting couple of hours to a day) in month.
The objective is to add additional consumers (worker) during these peak load duration and remove them when not being utilized (scaling up and down).

However it seems that one consumers in a consumer-group will read at-most one partition from the topic, thus the max number of consumers parallely processing the topic is limited to number of partitions in topic.

Is kafka suitable for these use cases where I would like to increase the number of worker 100 folds for a short bursts?

shanmuga
  • 4,329
  • 2
  • 21
  • 35
  • My first guess would be to create 100 partitions and then start 5 consumers handling 20 partions each. Then, when load comes, kill them and start 100 single-partition consumers. There is a good chance that some of existing tooling can handle all that automatically for your when you spawn more consumers, but you need to start with having a lot of partitions from very beginning. – Artur Biesiadowski Apr 16 '18 at 14:21
  • Won't kafka rebalance without killing the existing workers? Will it be possible increase the number of partition dynamically? – shanmuga Apr 16 '18 at 14:25
  • You can change amount of partitions on the fly: https://stackoverflow.com/questions/33677871/is-it-possible-to-add-partitions-to-an-existing-topic-in-kafka-0-8-2/33679300 - but it will not rebalance existing data and won't necessarily cause all producers to distribute new data across increased number of partitions automatically. This already goes outside of pure kafka and into specific frameworks you are using. – Artur Biesiadowski Apr 16 '18 at 14:30

0 Answers0