0

I am working on a spring boot cloud stream application that needs to read data from Azure Eventhub Namespace topic. I am trying to understand the relationship between Topic Partition count and spring.cloud.stream.bindings.input.consumer.concurrency.

If I will put partition count as 10 should I need to provide spring.cloud.stream.bindings.input.consumer.concurrency as 10 ? or there will be no impact if the value of these mismatch

Rohail Usman
  • 119
  • 3
  • 12

1 Answers1

1

There is a similar issue.

In short:

If you have 10 partitions, and then set the concurrency as 10. Then there will 10 threads to read data from the 10 partitions.

And if you define concurrency less than 10, then there is at least 1 thread to read data from 2 partitions. This will cause a little performance issue.

Ivan Glasenberg
  • 29,865
  • 2
  • 44
  • 60
  • Can this partition and concurrency mismatch result in much slowness? In my case I was reading a topic with 12 partitions but concurrency is set to 10 and the total data set is around 6500 but it took a day to complete the flow. – Rohail Usman Jan 11 '21 at 15:53
  • @RohailUsman, It will cause slowness. You'd better set it as 12 as well. – Ivan Glasenberg Jan 12 '21 at 01:04
  • that's for sure I will set it to twelve but I was confused with a smaller number of records, how come the process is so much slower with 10 concurrency levels. Is there any document that can help me understand it better? – Rohail Usman Jan 14 '21 at 11:50