1

Hello Stack Overflow community and anyone familiar with spring-kafka!

I am currently working on a project which leverages the @RetryableTopic feature from spring-kafka in order to reattempt the delivery of failed messages. The listener annotated with @RetryableTopic is consuming from a topic that has 50 partitions and 3 replicas. When the app is receiving a lot of traffic, it could possibly be autoscaled up to 50 instances of the app (consumers) grabbing from those partitions. I read in the spring-kafka documentation that by default, the retry topics that @RetryableTopic autocreates are created with one partition and one replica, but you can change these values with autoCreateTopicsWith() in the configuration. From this, I have a few questions:

  • With the autoscaling in mind, is it recommended to just create the retry topics with the same number of partitions and replicas (50 & 3) as the original topic?
  • Is there some benefit to having differing numbers of partitions/replicas for the retry topics considering their default values are just one?
Tucker Mullens
  • 33
  • 2

1 Answers1

2

The retry topics should have at least as many partitions as the original (by default, records are sent to the same partition); otherwise you have to customize the destination resolution to avoid the warning log. See Destination resolver returned non-existent partition

50 partitions might be overkill unless you get a lot of retried records.

It's up to you how many replicas you want, but in general, yes, I would use the same number of replicas as the original.

Only you can decide what are the "correct" numbers.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179