0

If producer throughput is 5 mb/sec and consumers is 15mb/sec , I want to achieve an overall throughput of 75 mb/sec. Is the right calculation rule for the number of partitions MAX(throughput/prod-throughput , throughput/consumer-throughput) ? or there are other considerations

Muhammad Magdi Youssif
  • 107
  • 6
  • 1
    Are you planing to use multiple producers/consumers? If 75mb/sec is your goal I guess you plan to run 15 producers and 5 consumers? I understand you ask about number of partitions, just trying to fully understand you first – Ofek Hod Mar 24 '20 at 15:39
  • And to follow up, Ofek's clarification suggests a minimum of 5 partitions, but as a general rule it is best to have more than the minimum to allow you to scale up the consumers if required without changing the number of partitions (which impacts affinity between keys and partitions). Producers doesn't affect the decision because any batch of published records will likely relate to several partitions so all producers will be writing to all partitions. – Chris Mar 29 '20 at 15:48

0 Answers0