1

I have 1 topic with K partitions, however the messages are of varying size. Let's say they have weights on a scale of 1 to 10, and the consumers take equivalent time/resources to process them.

How do we design a custom partitioner to spread the messages across the K partitions so that the K consumers have similar load?

mazaneicha
  • 8,794
  • 4
  • 33
  • 52
vrtx54234
  • 2,196
  • 3
  • 30
  • 53
  • Sounds like a bin packing problem https://en.wikipedia.org/wiki/Bin_packing_problem – mazaneicha Oct 26 '20 at 23:30
  • You could possibly do round robin partitioning for each message size. So for all messages of size 1, you do round robin, for all messages of size 2 you do round robin, etc. You would have to remember which partition the last message of size S has been published to. – user152468 Oct 27 '20 at 06:55
  • Partitioners are only for producers. Even if the producer equally distrbibuted the sizes, then the consumer assignment is not guaranteed to be equal as the consumer would not know the sizes of the messages until consuming and already processing them – OneCricketeer Oct 27 '20 at 15:50

0 Answers0