3

I would like to know the exactly meaning of "concurrency" when using spring.cloud.stream.default.consumer.concurrency property.

Documentation (https://docs.spring.io/spring-cloud-stream/docs/Chelsea.RELEASE/reference/html/_configuration_options.html) says "The concurrency of the inbound consumer" and this can be interpreted in several ways.

What kind of thread executor is created behind the scenes?

Thanks!

italktothewind
  • 1,950
  • 2
  • 28
  • 55

1 Answers1

7

The semantics of concurrency is dependent on the actual binder implementation. For example in the case of the Kafka binder, when you set concurrency, that value will be passed on to the underlying MessageListenerContainer. Lets say that your consumer application is consuming from a topic that has 3 partitions and you set the value of concurrency to 3. This will create 3 threads by the container in which each of them is handling a single partition from the topic (In fact, in the case of Kafka binder, 3 listener containers will be created behind the scenes). If there are more partitions on the topic than there are concurrent threads, then the partitions will be distributed across the various threads. The same semantics apply for the Rabbit binder as well although the actual implementation of it may vary slightly internally.

sobychacko
  • 5,099
  • 15
  • 26
  • Thanks. I have another doubt. Is there any warranty than no more than 3 threads will be created if concurrency = 3? Because in some cases another property, maxConcurrency, can be defined and this make me think that threads can grow over the defined concurrency threshold. – italktothewind Jan 19 '19 at 22:08
  • Please don't forget my last question :) – italktothewind Jan 22 '19 at 03:08
  • Hi, the maxConcurrency is a property that is binder specific. It is only supported in the Rabbit binder and not in the Kafka binder. If the maxConcurrency is greater than concurrentConsumers, then maxConcurrency will be used as the max number of concurrent consumers. You might want to review this method: https://github.com/spring-projects/spring-amqp/blob/master/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/listener/SimpleMessageListenerContainer.java#L176 – sobychacko Jan 22 '19 at 15:23
  • Hi @sobychacko, if kafta topic partition is 25 and if we set 'spring.cloud.stream.bindings.input.consumer.concurrency=20' in consumer service, what will happen, how many threads will be created in background? can yu pls share your thought on this – Priya Nov 25 '20 at 14:51
  • Hi, As I mentioned in the answer above if you have more partitions than concurrent threads, 25 partitions, but concurrency set to 20 for e.g, in that case, Spring Cloud Stream will create 20 threads and some threads will be consuming from more than one partition. Most of them will consume from a single partition, but there are a few that listen on more than one partition. – sobychacko Nov 25 '20 at 17:14
  • But in this case We ate talking about a single consumer application. What about if we have multiple instances of consumer application – user725455 Jun 29 '22 at 10:41
  • This SO thread is more than 3 years old. It might be beneficial if you can create a new question and maybe link this thread from that. – sobychacko Jun 29 '22 at 15:21