0

Consider this scenario: Kafka topic with 6 partitions. Spring Java Kafka Consumer Application with 6 replicas so that each of them deals with one of the partitions.

The problem I'm facing is the processing of each message in the consumer takes a long time (~20 seconds), since it needs to call a really slow external system.

So even though I've provisioned 6 partitions/replicas I end up having a bottleneck in which the 6 consumers block ~20 seconds per message, which means a throughput of 6 messages every 20 seconds!!

Could you suggest ways to speed up this scenario taking into account that I can't modify the behaviour of the external system?

codependent
  • 23,193
  • 31
  • 166
  • 308

1 Answers1

0

Increase the number of partitions and concurrency on each instance.

The number of partitions must be >= instances * concurrency.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks for you suggestion Gary. However I still think that wouldn't work it out. I've read it's not recommened to have a high number of partitions (https://www.linkedin.com/pulse/kafka-optimization-how-many-partitions-needed-maria-hatfield-phd/), with 10 max as a rule of thumb. So with 10 partitions and one processed message every 20 seconds I would have 30 messages per minute, which is way below my throughput requirements. What do you think? – codependent May 28 '20 at 16:23
  • 1
    Given your situation, I don't think you have a choice but to go with a larger number of partitions; anything else (such as asynchronously processing records from each partition) will add a great deal of complexity to your application. Why don't you just try it? – Gary Russell May 28 '20 at 16:31
  • Sure, I’ll give it a try increasing partitions. Any recommendation about the concurrency value per consumer instance? – codependent May 28 '20 at 16:48
  • 1
    It can probably be something quite large if the listener is blocking for such a long time; just make sure that `max.poll.records` and `max.poll.interval.ms` are set appropriately to avoid a rebalance. – Gary Russell May 28 '20 at 16:50