0

I have a kafka consumer class which has 2 listener TOPICS and a DLQ listener. When load testing with million records i.e.producing 15 requests per second that reaches topic-1 and 1 request per second reaches topic-2 simultaneously, I find that there is a lag in kafka consumption. Eg.if the 15 requests are produced at 2.00 am, the topic-1 listener is consuming only around 2.20am. Where as the Topic -2 that consumes 1 request per second does not show any time lag. Can someone please help me out on how to find the cause of this time lag and solution? Thank you in advance!

Please find the edited Simple Java code that illustrates this issue

Kafka consumerconfig.java

public class KafkaConsumer {
final ObjectMapper mapper =new ObjectMapper();
//MAIN TOPIC LISTENER 1
@KafkaListener(id = "topic-1", topics = "topic-1", groupId = "main", containerFactory= "kafkaListenerContainerFactory", clientIdPrefix = "topic-1")
    public void mainListener(ConsumerRecord<String, String> consumerRecord, Acknowledgement ack) {
dbService.saveTodb(consumerRecord.value(),mapper);
        ack.acknowledge();
 }

//MAIN TOPIC LISTENER 2
@KafkaListener(id = "topic-2", topics = "topic-2", groupId = "main", containerFactory= "kafkaListenerContainerFactory", clientIdPrefix = "topic-2")
    public void mainListener(ConsumerRecord<String, String> consumerRecord, Acknowledgement ack) {   
 dbService.saveTodb(consumerRecord.value(), mapper);
ack.acknowledge();
}
}

KafkaBeanFactory.java

@Configuration
public class KafkaBeanFactory{
@Bean(name = "kafkaListenerContainerFactory")
    public ConcurrentKafkaListenerContainerFactory<Object, Object> kafkaListenerContainerFactory(
            ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
            ConsumerFactory<Object, Object> kafkaConsumerFactory, KafkaTemplate<Object, Object> template) {
        ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
        configurer.configure(factory, kafkaConsumerFactory);
        var recoverer = new DeadLetterPublishingRecoverer(template,
                (record, ex) -> new TopicPartition("DLQ-topic", record.partition()));
        var errorHandler = new DefaultErrorHandler(recoverer, new FixedBackOff(3, 20000));
        errorHandler.addRetryableExceptions(JsonProcessingException.class, DBException.class);
        errorHandler.setAckAfterHandle(true);
        factory.setCommonErrorHandler(errorHandler);
        return factory;
}
}

application.yaml

spring:
 kafka:
    bootstrap-servers: localhost:9092 # sample value 
    client-id: mainDLQ
    properties:
      security:
        protocol: SASL_SSL
      sasl:
        mechanism: PLAIN
        jaas:
          config: org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="<string>";
          security:
            protocol: SASL_SSL
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      groupId: main
      enable-auto-commit: false
      auto.offset.reset: earliest
    listener:
      ack-mode: MANUAL_IMMEDIATE
KGT
  • 29
  • 3
  • 1
    It's hard to say; you would need to profile your application to see why one listener is slower than the other; perhaps one JSON object is more complex than the other; but I do have some observations - 1) you shouldn't be creating a new `ObjectMapper()` for each record; just reuse the same one each time (that would save some CPU and garbage collection). 2) You shouldn't have all 3 listeners in the same groupId (because a rebalance on one topic would cause an unnecessary rebalance on the others). – Gary Russell Mar 15 '23 at 18:56
  • Thank you Gary Russell for the quick response – KGT Mar 16 '23 at 16:11
  • I tried removing the DLT kafka configuration bean factory and just implemented 2 listeners. Now the consumtion is really fast.but i want DLT in place. Could you please suggest approaches without removing the DLT code? – KGT Mar 16 '23 at 16:29
  • 1
    It's not clear what you mean; the DLT listener is using the default factory (implicitly) and you have explicitly specified the default factory for the other two. It's hard to see why having a third listener (that's not running) would make any difference. – Gary Russell Mar 16 '23 at 16:34
  • Hi Gary.. I Tried removing the concurrentkafka listener bean containerfactory from my listener. NOW the consumption is fast with DLT. @KafkaListener(id = "topic-1", topics = "topic-1", groupId = "main", **containerFactory = "kafkaListenerContainerFactory", **clientIdPrefix = "topic-1") But is there any way in which i can make it fast without removing containerfactory? Need your suggestion – KGT Mar 27 '23 at 18:22
  • Don't put code in comments; edit the question instead and comment that you have done so. I don't see any way that that change would make a difference. Provide an [MCRE](https://stackoverflow.com/help/minimal-reproducible-example) so I can play with it myself. – Gary Russell Mar 27 '23 at 18:56
  • Thank you gary.. ive added the new code.. edited question – KGT Mar 27 '23 at 19:44
  • One more time; adding another listener (especially with `autoStartup="false"`) should not make any difference to the performance of the other two listeners). I need a minimal, complete simple project if you want me to investigate further. – Gary Russell Mar 27 '23 at 19:46
  • Thank you for your time and response @GaryRussell. Ive edited the question with a simple java code that has 2 listeners and a concurrentListenerFactory Configuration for your investigation. Please let me know if further details are requiredd. – KGT Mar 28 '23 at 12:17
  • Please post the complete, minimal, project someplace so I get exactly what you have and I don’t have to copy/paste. Include instructions on how to reproduce. – Gary Russell Mar 28 '23 at 12:21
  • @GaryRussell sure.. could please share a common place in which i can upload the code or your email id if any? – KGT Mar 29 '23 at 15:12
  • You can post it on GitHub or similar. – Gary Russell Mar 29 '23 at 15:39

0 Answers0