I am seeing issue whereby after some time, randomly, consumers stop consuming message from a topic and there's no error that was thrown. There's not many message in the topic because this is a test environment.
The consumer lag for that topic would be > 0 and would stuck at that number (like there's no consumer consuming it) indefintely. I inspected the topic using kafka tool, i can see that the partition is assigned to a consumer.
I'm running multiple threads of consumers consuming a topic, here is the code:
public Runnable getRunnable() {
return () -> {
final ReactiveKafkaConsumerTemplate kafkaConsumerTemplate =
kafkaLib.createKafkaConsumerTemplate();
kafkaConsumerTemplate.receive()
.concatMap(record -> {
// process message and commit offset
})
.subscribe();
};
}
@Override
public void run(ApplicationArguments args) {
ExecutorService executorService = Executors.newFixedThreadPool(2);
for (int i = 0; i < 2; i++) {
executorService.execute(getRunnable());
}
}
My question: if there're no message pushed to the topic for a period of time, would it be possible that kafkaConsumperTemplate.receive() would stop/exit/cancel subscription on its own? Or maybe the thread would exit somehow after subscription ended?
Using the reactive kafka consumer, there is no more @KafkaListener annotation so I'm using ApplicationRunner and spawn threads of consumers. Is there another alternative correct way of starting reactive kafka consumers ?
