0

We are trying to setup Kafka clusters using AWS MSK (Managed streaming for Kafka). We have multiple use cases where we may need >50 topics each having 5-25 partitions and cumulatively ~100 million events being produced in a single day.

Producer count - ~75

Consumer count - ~150

Taking cost into consideration (infrastructure and monitoring/maintenance)

Is it good to have multiple small clusters or few large shared clusters?

I understand some drawbacks of having large shared clusters where if one cluster goes down, it affects all topics, but maintenance and cost also play a big role where I feel having few large shared clusters might be preferable.

kushan85
  • 102
  • 1
  • 9
  • 1
    Given your estimates, there would be a max of 1250 partitions and around 1k events/sec. That doesn't sound too hard for a single cluster to handle. If you go separate clusters you have a big chance that each node won't be doing much and waste a lot of CPU/IO/RAM. – Augusto Feb 06 '22 at 20:27
  • Thank you @Augusto. Makes sense – kushan85 Feb 06 '22 at 20:33
  • There shouldn't be much maintenence cost when using MSK compared to self hosted – OneCricketeer Feb 07 '22 at 06:16
  • @OneCricketeer, there may still be maintenance in terms of setting up monitoring and alerts, upgrades, migrations etc. But I agree, with MSK, a lot of cluster maintenance will be take care of – kushan85 Feb 07 '22 at 22:39

0 Answers0