Kafka topic purge programmatically

Question

I tried to purge Kafka topic from Java as below by setting retention time to 1sec and then returning back to original value. But the the messages were not deleted from topic. What is wrong?

    Map<ConfigResource,Collection<AlterConfigOp>> altConf = new HashMap<ConfigResource,Collection<AlterConfigOp>>();
    Collection<AlterConfigOp> altConfOp = new ArrayList<AlterConfigOp>();
    AlterConfigOp aco1 = new AlterConfigOp(new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, "1000"), AlterConfigOp.OpType.SET);

    altConfOp.add(aco1);
    altConf.put(new ConfigResource(ConfigResource.Type.TOPIC, topic), altConfOp);
    
    ac.incrementalAlterConfigs(altConf);
    
    Thread.sleep(5000);

    altConf = new HashMap<ConfigResource,Collection<AlterConfigOp>>();
    altConfOp = new ArrayList<AlterConfigOp>();
    aco1 = new AlterConfigOp(new ConfigEntry(TopicConfig.RETENTION_MS_CONFIG, originalRetention), AlterConfigOp.OpType.SET);

    altConfOp.add(aco1);
    altConf.put(new ConfigResource(ConfigResource.Type.TOPIC, topic), altConfOp);
    
    ac.incrementalAlterConfigs(altConf);

The log cleaner is not guaranteed to have run within any time period — OneCricketeer, Oct 25 '21 at 13:05

score 0 · Answer 1 · answered Oct 25 '21 at 13:44

As mentioned in the comments, topic retention limits are lower bounds and Kafka does not guarantee when records over the limits will be deleted.

The proper way to delete records is to use the DeleteRecords API.

With the AdminClient, you can use deleteRecords(). This will delete all records with lower offsets than the offsets provided. You can use listOffsets() to find the end offsets of your partitions and pass them to deleteRecords() to effectively purge topics.

Kafka topic purge programmatically

1 Answers1