How to skip a msg that have error in kafka when i use ConcurrentMessageListenerContainer?

Question

I'm using spring-kafka-2.1.10.RELEASE try to skip the error data but failed,

it fall in a infinity consume loop

is there anything i mssing?

Appreciate if you can help

simple code:

=========================

@Service
public class testConsumerFactoryService {
    @Autowired
    private PlatformTransactionManager tx;
    @Autowired
    private TestRepository testRepository;  // table only have 1 column , String
    @Autowired
    private ConsumerFactory<String, String> consumerFactory;

    @PostConstruct
    public void consumerFactoryTest() {
      ContainerProperties containerProps = new ContainerProperties("test_1");
        containerProperties.setTransactionManager(this.tx);
        containerProperties
                .setMessageListener(new ConsumerAwareMessageListener<String, String>() {
                    @Override
                    public void onMessage(final ConsumerRecord<String, String> record,
                            final Consumer<?, ?> consumer) {
                        final String rec = record.value();
                        //add the 'rec' string to TestRepository's entity , aka: tests
                        try {
                            this.testRepository.save(tests);  //  >> 1. try to save in DB, but failed
                        } catch (final Throwable e) {
                            System.out.println("error !!" + e);
                            System.out.println("##records:" + record);
                            consumer.commitSync();  // >> 2. try to commit offset, but seems not working ?
                            System.out.println("##after:");
                        }
                    }
                }
            );
        final ConcurrentMessageListenerContainer<String, String> container = new ConcurrentMessageListenerContainer<>(
                this.consumerFactory, containerProperties);
        container.start();

    }
}

=============================== the config: ===============================

@Configuration
@EnableKafka
public class Config {
@Bean
ConcurrentKafkaListenerContainerFactory<String, String>
                    kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
                            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    //use AckMode.RECORD
    factory.getContainerProperties().setAckMode(AckMode.RECORD);
    return factory;
}

@Bean
public ConsumerFactory<String, String> consumerFactory() {
    return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}

@Bean
public Map<String, Object> consumerConfigs() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, embeddedKafka.getBrokersAsString());
    props.put(ConsumerConfig.GROUP_ID_CONFIG, 'Test1');
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
    return props;
}

First of all, welcome to SO. Can you post all of the `Config` class? You need to disable `ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG` in your consumerConfigs() method if you have not done it already. You should also check these posts: https://stackoverflow.com/a/60207989/5492826 and https://www.confluent.io/blog/spring-for-apache-kafka-deep-dive-part-1-error-handling-message-conversion-transaction-support/ if you need more in-depth error handling. — emrekgn, Apr 02 '21 at 12:07

score 0 · Accepted Answer · answered Apr 03 '21 at 21:56

0

In your ConcurrentKafkaListenerContainerFactory, just associate a SeekToCurrentErrorHandler:

factory.setErrorHandler(
        new SeekToCurrentErrorHandler(
            (rec, ex) -> {
              log.error(
                  "An exception happened during a message processing: {}", ex);
              // good to put something like a dispatch to a DLQ etc
            }));

With this, you will carry on to the next offset and you app won't get blocked due to a bad message etc.

answered Apr 03 '21 at 21:56

leozin

389
2
7

Thanks @leozin, seems there's no method called setErrorHandler() for the ConcurrentKafkaListenerContainerFactory – long_long_ago Apr 06 '21 at 02:53
Check this out: https://docs.spring.io/spring-kafka/docs/current/api/org/springframework/kafka/config/ConcurrentKafkaListenerContainerFactory.html you will see that it inherits the setErrorHandler method from AbstractKafkaListenerContainerFactory – leozin Apr 07 '21 at 11:58
hi @long_long_ago would that be possible upgrade your kafka client? I see that it was introduced on 2.2 and you are using 2.1 – leozin Apr 07 '21 at 12:34
thanks @leozin, i tried in version 2.6.7, it works(even dont set error handler &remove the commit , will goes in DefaultAfterRollbackProcessor), coz the version after 2.2 add in new features --> maxAttempts – long_long_ago Apr 08 '21 at 09:41

How to skip a msg that have error in kafka when i use ConcurrentMessageListenerContainer?

1 Answers1

Linked