We heavily use Kafka for our messaging needs(replaced MQ from our apps) and its one of the best decisions we made as it scales out easily as and when we need. We increase partitions when we know there will be new data coming in to the system. This doesnt have any impact to the kafka producer(metadata is refreshed time to time) or consumer(rebalance occurs when a new partition is added to a topic) that are used in our applications.
Recently, we had a requirement to group batch of records in a stream in 2minute window intervals and kafka streams looked like a perfect solution. However, after implementing it - we realized we cannot really scale out like we used to as every partition is tied to the changelog topic thats created for a state store.
And we cannot afford to create new topics every time or decide on a particular number of partitions that we need to start with.
This sounds like an enterprise issue. Any suggestions on this is highly appreciated!
