Get last event in Kafka - JAVA

Question

I'm actually using @KafkaListener to read events in a topic. I want to read 100 events, and after this call Thread.sleep() with a certain time period. My problem is, when the thread wakes up from sleep, the listener continues on the last event I read, but I want to discard the events when the thread is sleeping and continues with the last events in topic.

Like:

1-100 - Capture

Thread sleeping

101-500

Thread Returns

501 - 601 - Capture

The 101-500 events can be discarded

Code:

@KafkaListener(topics = "topic")
public void consumeBalance(ConsumerRecord<String, String> payload) throws InterruptedException {

    this.valorMaximoDeRequest = this.valorMaximoDeRequest + 1;
    if (this.valorMaximoDeRequest <= 100) {
        log.info("Encontrou evento )");
        log.info("Key: " + payload.key() + ", Value:" + payload.value());
        log.info("Partition:" + payload.partition() + ",Offset:" + payload.offset());

        JsonObject jsonObject = new Gson().fromJson(payload.value(), JsonObject.class);
        String accountId = jsonObject.get("accountId").getAsString();
        log.info(">>>>>>>>> accountId: " + accountId);

    } else {
        this.valorMaximoDeRequest = 0;
        Thread.sleep(60*1000);
    }

}

Kafka config:

    @Bean
    public Map<String, Object> kafkaFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put("specific.avro.reader", Boolean.TRUE);
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "brokers");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "1");

        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
        props.put("security.protocol", "SASL_PLAINTEXT");
        return props;
    }

score 0 · Answer 1 · answered Apr 05 '22 at 16:08

First, you shouldn't force the listening thread to sleep. The consumer may be considered as dead and trigger a consumer rebalance. You'd better use pause and resume on the consumer. See https://docs.spring.io/spring-kafka/docs/current/reference/html/#pause-resume

Then, if you want to skip the records published when the consumer was asleep, you'll have to seek (seekToBeginning) when the consumer awakes. See https://docs.spring.io/spring-kafka/docs/current/reference/html/#seek However it is not simple: The Kafka Consumer doesn't let you seek when the consumer is active nor when it does not own the partition.

score 0 · Answer 2 · answered Apr 10 '22 at 19:32

The point of having consumer groups is to keep track of the offsets which have been processed so that the subsequent consumption resumes from there and also distribute load across different consumers.

If your use-case doesn't need any of the above, you can use consumer.assign() which doesn't leverage the group management functionality.

@KafkaListener(topicPartitions = @TopicPartition(topic = "so56114299",
                      partitions = "#{@finder.partitions('so56114299')}"))
public void listen(@Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) String key, String payload) {
    System.out.println(key + ":" + payload);
}

Snippet reference: https://stackoverflow.com/a/56114699/2534090

Alternatively, you can write your own KafkaConsumer and manually call the consumer.assign() for assigning the partitions. To answer your original question, for seeking, you need to call consumer.seekToEnd() method every time your method wakes from Thread.sleep(). The subsequent poll will fetch the records from the end offset. Looks like you can add Consumer as parameter to your @KafkaListener method.

Get last event in Kafka - JAVA

2 Answers2