0

I have a use case similar to the one described here

which is that I would like to put the failed messages in a place where they can be retried at a pre-configured time interval (probably several minutes). Since I am not using Storm, the Kafka Spout is not an option as described in the accepted solution there. Is there another feature of Kafka that makes the message invisible to the consumers until the time period expires?

One of the goals of this project is to not write a scheduler (Cron or Java). Without the scheduler, the only other option is using JMS style messaging brokers. However, if Kafka has this functionality built-in I would like to use Kafka as we already have the infrastructure built for it.

Vijay Kumar
  • 2,439
  • 2
  • 32
  • 51
  • What about if your consumer wrote the failed message to another topic that you called a "retry" topic and then you just periodically tried the messages in that topic from the beginning? There's not an out of the box feature for this but it should be pretty simple to implement on the application side. – dawsaw Jan 13 '18 at 02:23
  • Thank you for your suggestion, this might work. Since a message could fail multiple times, would you suggest a separate topic for each retry. Like topics "retry-1", "retry-2","retry-n" etc. The problem with a single retry topic is that if I have processed all the messages in the whole topic before the retry interval expires, the message would be put on the retry topic and a consumer will try to process it immediately. – Vijay Kumar Jan 14 '18 at 22:37
  • Yeah that could make a lot of sense, especially with finite retries – dawsaw Jan 15 '18 at 00:23

0 Answers0