1

I have a Spring based service which subscribes Kafka topic. Service works fine but sometime service stops working with the following error. In such error scenarios, service stop subscribing topic and throws below error. If I give new consumer GroupID then service works fine again. I am doing manual Acknowledgment with Kafka. Please let me know what could be the issue. Why it needs new consumer groupID?

    Caused by: 
org.springframework.kafka.listener.ListenerExecutionFailedException: 
Listener method 'public void 
(org.springframework.messaging.Message<java.lang.String>,org.springframewo
rk.kafka.support.Acknowledgment) throws 
com.fasterxml.jackson.core.JsonParseException,com.fasterxml.jackson.databi
nd.JsonMappingException,java.io.IOException,java.lang.NumberFormatExceptio
n,java.lang.InterruptedException' threw exception; nested exception is 
java.lang.ArrayIndexOutOfBoundsException; nested exception is 
java.lang.ArrayIndexOutOfBoundsException

I have provided KafkaConfig details below.
Here is KafkaConfig:

   @Bean("kafkaListenerContainerFactory")
   @ConditionalOnMissingBean(name = "kafkaListenerContainerFactory")
   public ConcurrentKafkaListenerContainerFactory<String, String> listenerFactory() {
       ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
       factory.setConsumerFactory(consumerFactory());
       factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);       
        org.springframework.util.backoff.ExponentialBackOff backOff = new org.springframework.util.backoff.ExponentialBackOff(2000L, 1.5);  
        backOff.setMaxInterval(300000L);    
        factory.setErrorHandler(new SeekToCurrentErrorHandler(999999));       
       return factory;
   }



    @KafkaListener(topics = "${topics.kafkaInTopic}", containerFactory = "kafkaListenerContainerFactory") 
public void listen(Message<String> message, Acknowledgment acknowledgment) throws JsonParseException, IOException {

    try {
        //business logic
        acknowledgment.acknowledge();
       }
       Catch {
          //Error handling
          acknowledgment.acknowledge();

       }
}



    Property file:
    consumer:
      auto-offset-reset: latest
      enable-auto-commit: false
      jaas-config: 'org.apache.kafka.common.security.plain.PlainLoginModule  
      sasl-mechanism: PLAIN
      group-id:  sgroup1

Detailed Log:



2020-11-02T08:42:55.207-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1311)
2020-11-02T08:42:55.207-05:00 [APP/PROC/WEB/1] [OUT] ... 10 common frames omitted
2020-11-02T08:42:55.207-05:00 [APP/PROC/WEB/1] [OUT] Caused by: java.lang.ArrayIndexOutOfBoundsException: null
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] 2020-11-02 13:42:55.253 ERROR [,,,] 15 --- [ntainer#0-0-C-1] essageListenerContainer$ListenerConsumer : Error handler threw an exception
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.comp.esb.myService.listener.KafkaConsumerOrder.listen(org.springframework.messaging.Message<java.lang.String>,org.springframework.kafka.support.Acknowledgment) throws com.fasterxml.jackson.core.JsonParseException,com.fasterxml.jackson.databind.JsonMappingException,java.io.IOException,java.lang.NumberFormatException,java.lang.InterruptedException' threw exception; nested exception is java.lang.ArrayIndexOutOfBoundsException; nested exception is java.lang.ArrayIndexOutOfBoundsException
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:124)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1296)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1227)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1198)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1118)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:933)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:749)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:698)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] at java.lang.Thread.run(Thread.java:748)
2020-11-02T08:42:55.253-05:00 [APP/PROC/WEB/1] [OUT] Caused by: org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method 'public void com.comp.esb.myService.listener.KafkaConsumerOrder.listen(org.springframework.messaging.Message<java.lang.String>,org.springframework.kafka.support.Acknowledgment) throws com.fasterxml.jackson.core.JsonParseException,com.fasterxml.jackson.databind.JsonMappingException,java.io.IOException,java.lang.NumberFormatException,java.lang.InterruptedException' threw exception; nested exception is java.lang.ArrayIndexOutOfBoundsException; nested exception is java.lang.ArrayIndexOutOfBoundsException
Brijesh
  • 11
  • 2
  • Please show full stacktrace – OneCricketeer Nov 02 '20 at 23:57
  • You need to show the full stack trace. `JsonParseException` seems to indicate you have some badly formed JSON. – Gary Russell Nov 03 '20 at 14:50
  • Thank you for looking into it. Here is detail log. Please note if I provide new consumer groupID service works fine – Brijesh Nov 03 '20 at 15:09
  • @OneCricketeer, Thanks for looking into it. Log is provided in issue description. – Brijesh Nov 03 '20 at 15:23
  • @mike, Thank you. Topic retention time is 7 days. There could be some issue with consumer groupID and offset setting but I am not sure what it could be. – Brijesh Nov 03 '20 at 15:31
  • I think your issue is with `SeekToCurrentErrorHandler(999999)`. Changing the group id is simply skipping a bad message. Doesn't solve the problem of any new data also being bad – OneCricketeer Nov 03 '20 at 15:58
  • @OneCricketeer, Thanks. yeah, changing group id will skip old messages. I think messages are valid only because same topic/messages are subscribed in another environment (Stage) and works fine. Sporadically I am seeing this error. – Brijesh Nov 03 '20 at 18:36
  • Well, you can see SeekToCurrentErrorHandler in the stacktrace, so something is wrong there – OneCricketeer Nov 03 '20 at 21:05

0 Answers0