I'm using kafka framework for a transfer of logs from producer to consumer in a project. The log retention policies really confuse me. The global configs in the "server.properties" shows
log.retention.hours=168
I overrided this for a topic using the command
/opt/kafka/bin/kafka-configs.sh --zookeeper localhost:2181 --alter --topic topic1 --config retention.ms=1000
But the logs didn't get deleted after doing this. Then I've changed the global config in the "server.properties" file to
log.retention.ms=1000
And
log.segment.bytes=1073741824
And
log.retention.check.interval.ms=300000
I've restarted kafka but the logs aren't deleted even after 300 seconds So I reseted the retention time
log.retention.hours = 168
I also changed the retention time in dynamic config to 1hr
The next day, all the logs in the topic just disappared. Now the logs aren't getting stored even for 1 hour. The dynamic config shows
Topic: SyslogNATSrv PartitionCount: 1 ReplicationFactor: 1 Configs: cleanup.policy=delete,retention.ms=3600000,delete.retention.ms=3600000
Dynamic configs for topic SyslogNATSrv are: cleanup.policy=delete sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=delete, DEFAULT_CONFIG:log.cleanup.policy=delete} retention.ms=3600000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=3600000} delete.retention.ms=3600000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:delete.retention.ms=3600000, DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
And the "server.properties" shows
log.retention.hours=168
I don't understand why the logs are not getting stored even for 1 hour(not even 1 second). They should be stored for 1hour if it follows dynamic config or 7 days if it follows global config.
Please help me with this!
