4

In our design, Kafka topics are created on the fly and used for a short period of time. While new topics are created and used, previously created topics may go out of use. There is a need to periodically delete unused topics. Say for simplicity, we would like to delete all the topics that have not been used (and are empty) for last 2 days.

Is there an established solution for this case? Any pointers on how to achieve this?

(We will be using AWS MSK (Kafka version 2.8))

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Abhijat Alok
  • 41
  • 3
  • 1
    I think, you should implement an application to delete unused topics automatically. With kafka admin API and kafka metrics you can find unused topics. – J. Song Dec 07 '21 at 10:51
  • 3
    I'm not aware of a ready-to-use solution but one idea might be to check for consumer offsets of each topic periodically and decide if a topic is used or not based on its consumer offset changes in a specified time threshold? – emrekgn Dec 07 '21 at 10:59
  • 1
    "_topics are created on the fly and used for a short period of time_" - can you expand on your use-case/requirements? This doesn't sound like a common design pattern. – Robin Moffatt Dec 07 '21 at 12:12
  • What will verify the topics will never be used again? Deleting "unused" topics is no different than deleting any other – OneCricketeer Dec 07 '21 at 14:20
  • 1
    @RobinMoffatt : The consumers when ready create UUID topics and start listening to it. They may go away unannounced and its ok for messages left in the queue to time out. This creates a need for topics to be purged on the server periodically. – Abhijat Alok Dec 08 '21 at 11:01
  • You can the UUID to message key instead to the topic name so consumers when ready can filter the messages by UUID. – Luis Estrada Nov 01 '22 at 14:32

0 Answers0