Pause and Resume KafkaConsumer

Question

What I've to do is pause the KafkaConsumer if during message consuming an error is thrown.

This is what I wrote

@KafkaListener(...)
public void consume(
 @Header(KafkaHeaders.CONSUMER) KafkaConsumer<String,String> consumer,
 @Payload String message) {
  
    try {
        //consumer message
    } catch(Exception e) {
        saveConsumer(consumer);
        consumer.pause();
    }
}

Then I wrote a REST service in order to resume the consumer

@RestController
@RequestMapping("/consumer")
class ConsumerRestController {
    @PostMapping("/resume")
    public void resume() {
        KafkaConsumer<String,String> consumer = getConsumer();
        if(consumer != null) {
            consumer.resume(consumer.paused());
        }
    }
}

Now, I've two questions. First question: When I call consumer.pause() from @KafkaListener annotated method what happens? Consumer is immediately paused or I can receive other messages associated on other offset of same topic-partition. For example, I have "message1" with offset 3 and "message2" with offset 4, "message1" cause an exception, what happens to "message2"? Is it consumed anyway?

Second question: Resuming the consumer from REST service give a ConcurrentModificationException because KafkaConsumer is not thread safe. So, how come I have to do this?

score 3 · Answer 1 · answered Oct 19 '20 at 15:09

3

Do not pause the consumer directly; pause the container instead.

@KafkaListener(id = "foo", ...)

@Autowired KafkaListenerEndpointRegistry;

...

registry.getListenerContainer("foo").pause();

The pause will take effect before the next poll; if you want to immediately pause (and not process the remaining records from the last poll), throw an exeption after pausing (assuming you are using the, now default, SeekToCurrentErrorHandler.

answered Oct 19 '20 at 15:09

Gary Russell

166,535
14
146
179

Thanks. For resuming instead? I need to resume and seek the consumer according a custom logic. – Vin Oct 19 '20 at 15:35
However I do not fully understood what is the difference between pausing the container and pausing the consumer. What would happen if `KafkaListener` is configured in order to listen on two or more topic or consumers listen on different partitions? I would pause/resume only the consumer who cannot process correctly the message, not all consumers. – Vin Oct 19 '20 at 15:43
Resume the container using the endpoint registry, the same way; if you need to perform seeks before resuming your listener should extend `AbstractConsumerSeekAware`. https://docs.spring.io/spring-kafka/docs/2.6.2/reference/html/#seek – Gary Russell Oct 19 '20 at 15:45
`@KafkaListener` is `@Repeatable`; if you want to control the consumers independently, you should use multiple annotations. – Gary Russell Oct 19 '20 at 15:47
Unfortunately this cannot help me. I cannot use multiple annotations. What I continue to not understand is why `KafkaConsumer` as that logic, I see the same thread can invoke its methods (a part `wakeup`). Doc says `KafkaConsumer` is not thread safe, but instead seems it not support multithreading at all. – Vin Oct 20 '20 at 08:34
That is just the way the kafka-clients `KafkaConsumer` works; the only way to achieve concurrency is by using multiple consumers; which is exactly what the container does via its `concurrency` property. – Gary Russell Oct 20 '20 at 13:31

Pause and Resume KafkaConsumer

1 Answers1