Partition Count override

Question

From here https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/master/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/KafkaMessageChannelBinder.java it seems is not possible, but asking anyway in case I missed it.

In Spring Cloud Data Flow, trying to create a flow, but need to set custom partitions for a topic.

It seems the partition count is calculated based on the instance count of the apps. Is there any way to override the numbers of partitions used on a topic?, they would have to be set on both sides of the queue(consumer, producer) but that would not be an issue.

Thanks in advance

Answer 1

You can override the default behavior by setting: --spring.cloud.stream.kafka.binder.minPartitionCount=<CUSTOM_VALUE>.

Please pay attention to prerequisites described for this property from the reference guide.

Also, you will find the following NOTE from the guide generally useful.

"The Kafka binder uses the partitionCount setting of the producer as a hint to create a topic with the given partition count (in conjunction with the minPartitionCount, the maximum of the two being the value being used). Exercise caution when configuring both minPartitionCount for a binder and partitionCount for an application, as the larger value is used. If a topic already exists with a smaller partition count and autoAddPartitions is disabled (the default), the binder fails to start. If a topic already exists with a smaller partition count and autoAddPartitions is enabled, new partitions are added. If a topic already exists with a larger number of partitions than the maximum of (minPartitionCount or partitionCount), the existing partition count is used."

As for SCDF, you can override this property for App either from the Shell or UI. If you want this to be globally applied to all the Stream Apps, that's possible as well - see here.