0

Here is the thing - I have to read a message from Kafka and check if some specific header has date >= than now ---> then handle it accordingly, otherwise I should be able to get the same message as many times as needed for the condition above to become true. Off course another messages in the same topic shouldn't be blocked. How can I do that? What I have in my mind so far is to republish the same message again to the kafka as many times as needed to meet the condition. But is that approach good? Wouldn't it simply duplicate messages in kafka and make the storage of kafka locked?

Jun
  • 21
  • 5

1 Answers1

0

should be able to get the same message as many times as needed for the condition above to become true

You can. Disable consumer group auto commits. Consume the topic in a loop and use seek functions to repeat from some offset. Break the loop and commit the offset, if you find the data you want.

other messages in the same topic shouldn't be blocked

They never will be

republish the same message again... Would it make the storage of kafka locked?

Kafka deletes old segments after a week, so no, not unless you fill up the disks before then, causing the server to crash.

Ultimately, this isn't a good use case for Kafka. Dump your data into some time series database, then query it from there.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • thanks for your answer. Currently I'm using error manual throwing/handling if the condition is not met. I just manually throw specific error until I meet the condition and skip handling that error in the catch block. Otherwise I commit offset manually. Is this approach suitable with kafka? – Jun Apr 27 '23 at 15:07
  • This is more custom consumer logic, and not really a "kafka problem", so you can do whatever you want as long as it works. I still think querying an actual database, indexed by time, would be a better option that doesn't involve much error handling or retries – OneCricketeer Apr 27 '23 at 15:47