Destination resolver returned non-existent partition

Question

I am using Spring-Kafka to consume messages from Confluent Kafka and I am using RetryTopicConfiguration Bean to configure the topics and backoff strategy. My application works fine but I see a lot of WARNING log like the one below in my logs and I am wondering if my configuration is incorrect.

DeadLetterPublishingRecovererFactory$1 : Destination resolver returned non-existent partition flow-events-retry-0-4, KafkaProducer will determine partition to use for this topic

Config Code

@Bean
    public KafkaTemplate kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

    @Bean
    public RetryTopicConfiguration myRetryableTopic(KafkaTemplate<String, Object> template) {
        return RetryTopicConfigurationBuilder
                .newInstance()
                .exponentialBackoff(BACKOFF_INITIAL_DELAY_10MINS, BACKOFF_EXPONENTIAL_MULTIPLIER_3, BACKOFF_MAX_DELAY_4HRS)
                .maxAttempts(5)
                .doNotAutoCreateRetryTopics()
                .setTopicSuffixingStrategy(TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE)
                .create(template);
    }

The retry topics are created separately with 1 partition and replication factor of 3.

score 2 · Accepted Answer · answered Jan 24 '22 at 15:11

By default, the same partition as the original topic is used; you can override that behavior by overriding the DeadLetterPublishingRecovererFactory @Bean:

@Bean(RetryTopicInternalBeanNames.DEAD_LETTER_PUBLISHING_RECOVERER_FACTORY_BEAN_NAME)
DeadLetterPublishingRecovererFactory factory(DestinationTopicResolver resolver) {
    DeadLetterPublishingRecovererFactory factory = new DeadLetterPublishingRecovererFactory(resolver) {

        @Override
        protected TopicPartition resolveTopicPartition(ConsumerRecord<?, ?> cr, DestinationTopic nextDestination) {
            return new TopicPartition(nextDestination.getDestinationName(), -1); // Kafka Chooses
//          return new TopicPartition(nextDestination.getDestinationName(), 0);  // explict
        }

    };
    factory.setDeadLetterPublishingRecovererCustomizer(dlpr -> {
        // ...
    });
    return factory;
}

As you can see in this example, you can also customize DLPR properties here too.

/**
 * Creates and returns the {@link TopicPartition}, where the original record should be forwarded.
 * By default, it will use the partition same as original record's partition, in the next destination topic.
 *
 * <p>{@link DeadLetterPublishingRecoverer#checkPartition} has logic to check whether that partition exists,
 * and if it doesn't it sets -1, to allow the Producer itself to assign a partition to the record.</p>
 *
 * <p>Subclasses can inherit from this method to override the implementation, if necessary.</p>
 *
 * @param cr The original {@link ConsumerRecord}, which is to be forwarded to DLT
 * @param nextDestination The next {@link DestinationTopic}, where the consumerRecord is to be forwarded
 * @return An instance of {@link TopicPartition}, specifying the topic and partition, where the cr is to be sent
 */
protected TopicPartition resolveTopicPartition(final ConsumerRecord<?, ?> cr, final DestinationTopic nextDestination) {
    return new TopicPartition(nextDestination.getDestinationName(), cr.partition());
}

Hi Gary, I've tried to implement your answer but I'm getting this error saying that DestinationTopicResolver bean could not be found. — Rafa Acioly, Mar 08 '22 at 17:35
Please ask a new question showing your code and configuration. — Gary Russell, Mar 08 '22 at 20:14
@GaryRussell In `spring-kafka 3.0.6`, if `-1` is used how does the producer assign a partition to the record? Will it use the same key that was used to publish the original topic? — tuk, Jul 10 '23 at 17:20
See the javadocs for the default assignor: https://kafka.apache.org/documentation/#consumerconfigs_partition.assignment.strategies. — Gary Russell, Jul 10 '23 at 20:21

Destination resolver returned non-existent partition

1 Answers1

Linked