Confluent Kafka: Setting Retention.ms for an individual topic not working as expected

Question

I'm using confluent Kafka in my project where the messages sent to a particular topic need to be deleted after a retention time. So I set retention.ms for the individual topics but it's not working (Still I can see messages after its retention time)

I have browsed most of the stack questions but still, I can't able to find a proper reason/ solution for Kafka retention.ms not working problem.

I followed the below steps to create and set retention time in ms.

1. Created a topic say. 'user_status'

2. updated its retention.ms time by following below code

    from confluent_kafka.admin import AdminClient, ConfigResource, NewTopic, NewPartitions
    from confluent_kafka import Producer, Consumer, KafkaError, KafkaException
   
    topic_config = ConfigResource('topic', 'user_status')
    topic_config.set_config('retention.ms', '5000')
    admin.alter_configs([topic_config])
   

3. Send message from producer end.

4. waited for 6000 ms time.

5. Tried to receive the message from particular topic. But I received the message, And still the message was not deleted by retention policy.

Note: I ensured below After updating retention.ms I verified the same has been updated in kafka topic information (Topic describe).

Also, I updated server.properties with the log.retention.check.interval.ms=1 ms and restarted Kafka service after I updated the properties file.

what I expect from the above question

I want to set a retention.ms to an individual topic and the message passing that time should be automatically deleted as defined in Kafka policy.

what happening now with my current code. The messages are still received by consumer even after retention.ms time.