How to set change the retry attempts based of the exception type in Kafka SeekToCurrentErrorHandler?

Question

I have a spring boot application and i am trying to create a Kafka retry listener wherein, based of the exception type i need to change the retry attempts.

For example:

If exception type is A then the retry attempts should be 5
If exception type is B then the retry attempts should be 10

Can anyone recommend how do to this in Spring Kafka?

Following is my ListenerFactory. I am using SeekToCurrentErrorHandler

Bean
    public ConcurrentKafkaListenerContainerFactory<String, test> retryListenerContainerFactory() {

        ConcurrentKafkaListenerContainerFactory<String, test> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
        SeekToCurrentErrorHandler errorHandler = new SeekToCurrentErrorHandler((record, exception) -> {
            System.out.println(
                    "RetryPolicy** limit has been exceeded! You should really handle this better." + record.key());
        }, new FixedBackOff(0L, 3L));
        errorHandler.addNotRetryableException(IllegalArgumentException.class);
        errorHandler.setCommitRecovered(true);
        factory.setErrorHandler(errorHandler);
        // to keep the consumers alive the failure gets reported to broker so that
        // consumers remain alive
        factory.setStatefulRetry(true);
        factory.setConcurrency(2);
        return factory;
    }

score 2 · Answer 1 · answered Nov 06 '20 at 14:34

2

Starting with version 2.6, you can add a function to determine the BackOff to use, based on the consumer record and/or exception:

/**
 * Set a function to dynamically determine the {@link BackOff} to use, based on the
 * consumer record and/or exception. If null is returned, the default BackOff will be
 * used.
 * @param backOffFunction the function.
 * @since 2.6
 */
public void setBackOffFunction(BiFunction<ConsumerRecord<?, ?>, Exception, BackOff> backOffFunction) {
    this.failureTracker.setBackOffFunction(backOffFunction);
}

This is only called when there is no current BackOff for this record (i.e. it won't help with your other question).

answered Nov 06 '20 at 14:34

Gary Russell

166,535
14
146
179

I opened an issue to allow resetting the failure tracker if the exception type changes: https://github.com/spring-projects/spring-kafka/issues/1610 – Gary Russell Nov 06 '20 at 19:36
Thanks alot @GaryRussell !! I have another question, is there a way i can set the no. of retry attempts with ExpoentialBackOff? My requirement is to change the no. of retries based of the exception type, and also cause a delay exponentially between each retry. Can you please suggest. For now i am doing this. errorHandler.setBackOffFunction((record,exception)-> { if(msg.getNewError().equals(errorCodes.getApiError()) return new FixedBackOff(0L, apiRetryCount); else return new FixedBackOff(0L, retryCount ); }); – user1945064 Nov 07 '20 at 06:57
You are using `FixedBackOff` with no delay between retries. Use an `ExpontialBackOff` instead - `new ExponentialBackOff(initial, multiplier)`. You can also set the max interval to add a cap to the delay between attempts, and `setMaxElapsedTime` to stop retrying (instead of a retry count) (you can calculate that based on the retry count and the other settings). 2.6.3 with the new feature will be released next week, – Gary Russell Nov 07 '20 at 18:49
Thankyou. Have another question. In the recovery when all the retries are exhausted i need to put the message to error topic. can i use DealLetterPublishingRecoverer for that purpose? – user1945064 Nov 09 '20 at 23:25
Yes. That is what it is for. – Gary Russell Nov 10 '20 at 00:54
Thankyou but my requirement is that once all the retries are exhausted, i need to save the record to the database... and if database is down or any exception gets thrown while saving to database then it should go to a error topic. Can you please suggest. SeekToCurrentErrorHandler errorHandler = new SeekToCurrentErrorHandler((record, exception) -> { try { //when all retries are exhausted save to database and if exception is thrown then put to deadletter topic } catch(Exception se) { //new DeadLetterPublishingRecoverer(getEventKafkaTemplate()); } }); – user1945064 Nov 10 '20 at 07:44
Don't put code in comments; it doesn't render well and is hard to read; edit the question instead and comment that you have done so. Don't create a new recoverer in the `catch` block; simply define it elsewhere, say in variable `dlpr` and call `dlpr.accept(record, exception);` if the DB store fails. – Gary Russell Nov 10 '20 at 13:58

How to set change the retry attempts based of the exception type in Kafka SeekToCurrentErrorHandler?

1 Answers1