7

For a topic with multiple partitions -

1) Does a single SpringBoot instance use multiple threads to process (method annotated with StreamListener) each message from each partition?

2) Is it possible to configure more than one thread for each partition or is that something I would have to manually hand off from my listener thread to a worker pool?

Ajay
  • 465
  • 1
  • 9
  • 22

1 Answers1

6

....consumer.concurrency controls the number of threads (default 1).

The partitions are distributed across the threads. If you have 20 partitions and 4 threads; they'll get 5 partitions each.

You need to have at least as many partitions as the aggregate concurrency across all instances. (If you have 2 app instances and 5 threads each, you need at least 10 partitions).

You should not distribute messages from a single partition across multiple threads; the offset will be committed as soon as you hand off to the new thread and that could cause message loss.

You should always err on the side of having more partitions than you need concurrency.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • 2
    To add to Gary's comment, here's a good article that should guide you through the sizing process.https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ – Marius Bogoevici Feb 05 '17 at 16:35