When we decide about partitions, should we do that on per-topic base, or it is topic-wide decision? If T1 partitioned on 3 partitions, and T2 partitioned on 2 partitions, can they both be consumed by 1 consumer?
Or it is better make equal number of partitions if topics must be consumed by 1 consumer?
I ask that because high level consumer can be created by passing topics and partition number.
So I wonder should I pass to that constructor only topics with equal partition number?
When we create high level consumer, we pass not partition number, but intended number of consuming threads(streams).
The answer is yes, they can be consumed by 1 consumer. (If that consumer subscribed to both topics) Consumer just opens N streams/intended number of consuming threads (you pass that as a parameter!). If N < P(number of all partitions of all topics), then some streams will collect data of several partitions. If N > P, some streams will be in non-busy wait.
It is desirable to have P=N, but it is even better to have N > P , because tomorrow if new partitions appear - you will be ready for grater load.
I've done a research on that and created a blog entry