3

We use reactiveKafkaConsumerTemplate to receive messages,then process the message. We try to graceful shutdown the application.

  1. Stop poll new messages
  2. Wait for the current message finish process
  3. Shutdown application

We tried reactiveKafkaConsumerTemplate.pause. It doesn’t stop consumer poll new messages when there is back pressure pause. We also tried dispose, it doesn't wait for message complete processing. How could we archive the graceful shutdown? Thanks.

log
r.k.r.internals.ConsumerEventLoop - Paused - back pressure
ReactiveKafkaConsumerTemplate pause topic=test-topic, partition=5
r.k.r.internals.ConsumerEventLoop - Paused - Consumer woken
r.k.r.internals.ConsumerEventLoop - Paused - Resume

reactiveKafkaConsumerTemplate.
receiveAutoAck()
.publishOn(Schedulers.boundedElastic())
.flatMap(x -> Mono.just(x)
          .delayElement(Duration.ofMillis(300)),5)
.flatMap(message -> Mono.just(message)
       .flatMap(processMessageImp::processMessage)
       .onErrorResume(t -> Mono.empty())
);


public void pauseKafkaMessageConsumer() {
 reactiveKafkaConsumerTemplate.assignment()
     .doOnNext(tp -> log.info("ReactiveKafkaConsumerTemplate pause topic={}, partition={}",
      tp.topic(),tp.partition()))
     .flatMap(topicParts -> reactiveKafkaConsumerTemplate.pause(topicParts))
     .subscribe();

}

@PreDestroy
public void onExit() {
        pauseKafkaMessageConsumer();
try {
  Thread.sleep(5000);
} catch (InterruptedException e) {
 log.error("onExit Error while PreDestroy ");
}

}

with 1.3.13 consumer paused in back pressure case, But Consumer resumed after rebalance, Would it be expected behavior?

12/8/22 10:53:42.559 PM DEBUG r.k.r.internals.ConsumerEventLoop Emitting 1 records, requested now 1
12/8/22 10:53:42.559 PM DEBUG r.k.r.internals.ConsumerEventLoop - onRequest.toAdd 1, paused false
12/8/22 10:53:45.966 PM DEBUG r.k.r.internals.ConsumerEventLoop Async committing: {test-topic-9=OffsetAndMetadata{offset=10934, leaderEpoch=null, metadata=''}}
12/8/22 10:53:46.258 PM reactiveKafkaConsumerTemplate pause
12/8/22 10:54:06.185 PM DEBUG r.k.r.internals.ConsumerEventLoop onPartitionsRevoked [test-topic-9, test-topic-8]
12/8/22 10:54:07.289 PM DEBUG r.k.r.internals.ConsumerEventLoop onPartitionsAssigned [test-topic-9, test-topic-8]
12/8/22 10:54:07.505 PM DEBUG r.k.r.internals.ConsumerEventLoop Emitting 12 records, requested now 1

Gin
  • 31
  • 2
  • What version are you using? The latest is 1.3.14; 1.3.13 fixed a problem where pausing while already paused for back pressure didn't work. https://github.com/reactor/reactor-kafka/releases – Gary Russell Nov 21 '22 at 14:04
  • we are using 1.3.13. – Gin Nov 21 '22 at 17:29
  • Please provide an [MCRE](https://stackoverflow.com/help/minimal-reproducible-example) and I will take a look. – Gary Russell Nov 21 '22 at 19:04
  • Looks like a logging error only; the `Resumed` log is emitted even if no partitions are actually resumed (I just tested it). The code in the event loop looks like this `toResume.removeAll(ConsumerEventLoop.this.pausedByUser);` `toResume` ends up empty but the resumed log is still emitted. But I don't see any more emit messages. – Gary Russell Nov 21 '22 at 20:00
  • Double checked. at runtime, the lib overridden by 1.3.11. We’ll update the version at runtime and try the same test. Thanks! – Gin Nov 21 '22 at 21:39
  • Consumer paused on back pressure case after runtime update to version 1.3.13 – Gin Nov 22 '22 at 19:21
  • Consumer resumed after rebalance, Would it be expected behavior? – Gin Dec 09 '22 at 21:51
  • There is logic to not resume topics paused due to back pressure, but the logic for user pauses is missing; please open an issue on GitHub. – Gary Russell Dec 09 '22 at 22:33
  • https://github.com/reactor/reactor-kafka/issues/305 Thanks! – Gin Dec 09 '22 at 23:29
  • We upgraded the reactor-kafka version to 1.3.15. Now it works as 1.Partition assigned to consumer1 is 4,6,5,7 2.User pause consumer 3.Rebalance happen, a.partition 4,5,6,7 revoked b.Partition 6,7,8,9,10,11 assigned to consumer1 c. Per commit log, partition 6,7 might be paused. Consumer wake up and consume from partition 8,9,10,11 We expected the consumer pause for all partitions after rebalancing. – Gin Dec 16 '22 at 00:21
  • 1
    The framework tracks exactly which partitions the user code paused; it does not know if that represents all the previously assigned partitions, or some subset. Even if it did know it was the previous full assignment, it can't make any assumptions about the desired behavior when a different subset of partitions is assigned; it can only re-pause the partitions it knows about. I added a new feature request: https://github.com/reactor/reactor-kafka/issues/307 – Gary Russell Dec 16 '22 at 13:54
  • Thanks! Looking for the enhancement to be available. – Gin Dec 19 '22 at 19:39

0 Answers0