4

My application is receiving messages using Spring Cloud Stream bound to GCP pubsub. I'm experimenting with the following configuration parameters:

spring.cloud.gcp.pubsub.subscriber.executor-threads
spring.cloud.stream.bindings.<channelName>.consumer.concurrency

The application has 3 different channels, each of which has a consumer group defined. Multiple instances of the application will be running in production (on kubernetes).

I'm trying to find the right settings to configure to determine how many messages can be processed in parallel in each app instance. I've been experimenting on my local machine tweaking both of the above parameters, but only executor-threads seems to have any effect. If I set it to say 5, and pump a bunch of messages into the system, I see 5 threads in my message handling logs. If I bump it up to 10, I see 10 threads there. The concurrency parameter, however, doesn't seem to do anything, no matter if it's set to 1 or 10 or whatever.

What is the relationship between these parameters, if any? Is the concurrency parameter only used for other binders like Rabbit or Kafka?

Thanks.

Dave McCullough
  • 43
  • 2
  • 5

1 Answers1

7
spring.cloud.gcp.pubsub.subscriber.executor-threads
spring.cloud.gcp.pubsub.subscriber.parallel-pull-count

These two configuration options in Spring Cloud GCP are identical to setExecutorThreadCount and setParallelPullCount in Pub/Sub's Java client library. They control the behavior of your Pub/Sub subscriber client.

parallel-pull-count determines how many streams your subscriber client opens to receive messages and executor-threads determines how many threads your subscriber client uses to process message callbacks (where messages are acknowledged so the Pub/Sub service does not send them to you again). For example, when you set parallel-pull-count to 2 and executor-threads to 4, messages will come in via 2 streams and 8 threads will be busy processing them. See Concurrency Control for more information.

spring.cloud.stream.bindings.<channelName>.consumer.concurrency

You are right that this configuration option is only for certain binders. In order to set concurrency in your consumer groups, your producer must be partitioned. However, the Spring Cloud Stream binder to Pub/Sub currently does not support partitioning, thus you will not be able to set concurrency.

I'm trying to find the right settings to configure to determine how many messages can be processed in parallel in each app instance.

To achieve maximum throughput, I would play with the first two configuration options above, while also keeping in mind that messages that take longer to process than their acknowledgement deadline become expired and the Pub/Sub service will redeliver them.

tianzi
  • 185
  • 1
  • 7