Consumer is consuming same message multiple time

Question

I have an Kafka consumer that consumer message . I have set retry template in KafkaListenerContainerFactory . But don't know why it is consuming the same message twice . When the application through exception retry template called with the specified count also the Kafka consume the same message with the same count.

@Bean
RetryTemplate retryTemplate() {
    RetryTemplate retryTemplate = new RetryTemplate();

    FixedBackOffPolicy fixedBackOffPolicy = new FixedBackOffPolicy();
    fixedBackOffPolicy.setBackOffPeriod(1000L);
    retryTemplate.setBackOffPolicy(fixedBackOffPolicy);

    SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
    retryPolicy.setMaxAttempts(3);
    retryTemplate.setRetryPolicy(retryPolicy);

    return retryTemplate;
}


@Bean("KafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<String, String> KafkaListenerContainerFactory(RetryTemplate retryTemplate) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setRetryTemplate(retryTemplate);
    factory.setRecoveryCallback(context -> {
        log.error("Maximum retry policy has been reached");
        return null;
    });
    factory.setConcurrency(Integer.parseInt(kafkaConcurrency));
    return factory;
}

Consumer

@KafkaListener(topics = "${kafka.topic.json}", containerFactory = "kafkaListenerContainerFactory")
public void recieveSegmentService(String KafkaPayload) throws Exception {
    KafkaSegmentTrigger kafkaSegmentTrigger;
    kafkaSegmentTrigger = TransformUtil.fromJson(KafkaPayload, KafkaSegmentTrigger.class);
    log.info("Trigger recieved from segment service {}", kafkaSegmentTrigger);
    try {
        processMessage(kafkaSegmentTrigger);
    } catch (Exception e) {
        retryTemplate.execute(arg0 -> {
            processMessage(kafkaSegmentTrigger);
            return null;
        });
    }finally {
    }
}

The processMessage is throwing the exception

score 0 · Accepted Answer · answered Apr 02 '19 at 19:59

0

You have nested RetryTemplates - one outside the listener and one inside. If you are using the same template is both places, you'll get 12 attempts (3 driven by the listener adapter x4 inside the listener).

Use one or the other.

answered Apr 02 '19 at 19:59

Gary Russell

166,535
14
146
179

It's actually 12 attempts. – Gary Russell Apr 02 '19 at 20:01
Thanks sir..So should i remove either setting retry template in Kafka consumer factory or i remove retryTemplate.execute method in Listener – Bharat Wadhwa Apr 03 '19 at 05:56
If i remove..retryTemplate.execute and how i will acknowledge offset ..If i add that in finally then it will be execute every time – Bharat Wadhwa Apr 03 '19 at 06:30
The listener container will commit the offset when retries are exhausted. – Gary Russell Apr 03 '19 at 12:15
@Gray Thanks for helping out.. In listener i haven't acknowledge manually...does it perform this commit internally – Bharat Wadhwa Apr 03 '19 at 12:26
The listener container commits the offset when the listener exits normally, which it will do because the recovery callback returns normally after retries are exhausted. – Gary Russell Apr 03 '19 at 12:47
But if i am manually committing the offset in that case i have to commit it manually after the retry gets exhausted.. – Bharat Wadhwa Apr 03 '19 at 12:59
If you are using manual acks; do the ack in the recovery callback - see [my answer to your other question](https://stackoverflow.com/questions/55493989/how-to-set-acknowledgement-in-case-when-retry-gets-exhausted-in-kafka-consumer/55495482#55495482). – Gary Russell Apr 03 '19 at 13:05

Consumer is consuming same message multiple time

1 Answers1