Kafka - How to move expired messages to different topic?

Question

The requirement is to have a Kafka topic accept messages. But the message has to be consumed and acknowledged within a certain time. If not another process should pick up the message and start a different process (say abort or rollback process).

One approach is to move the message to a different topic after the time elapsed. And a different consumer can listen to it and start the abort process. How is this possible in Kafka? Or is there a different approach available for this?

score 0 · Answer 1 · answered Apr 08 '21 at 07:51

0

You can get the timestamp of the message in topic and decide what to do based on it:

@KafkaListener(id = "groupId", topics = "topic1")
public void listen(String msg, @Header(KafkaHeaders.RECEIVED_TIMESTAMP) long ts) {
    if(ts > currentTs - validGapTs){
        //time stamp is valid. So, do your stuff
   }
   else {
        // send message to another topic to rollback
   }
}

answered Apr 08 '21 at 07:51

Hamed

2,084
6
22
42

But by the time the listen function get executed, the time may not elapsed. This scenario happens when the actual consumer for that message is unavailable. So the message needs to wait in the queue for some time and move to a different topic. – Jawahar Apr 08 '21 at 08:12

score 0 · Answer 2 · answered May 23 '21 at 13:28

The brokers have no mechanism for this

You need to consume the message in order to know how much time has elapsed.

Assuming, cleanup.policy=delete, if you don't consume the data within the retention window, the event will be unavailable

The alternative would have a compacted topic create a table of events that acts as your processing status of pending/processing/processed/aborted. Periodically get all pending tasks and check if their timestamps fall out of the window that you expect

Kafka - How to move expired messages to different topic?

2 Answers2