0

on all our kafka machines ( production machines ) , we see that: ( no free space )

df -h /var/kafka 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         11T   11T  2.3M 100% /var/kafka

and under /var/kafka/kafka-logs

we see all topic folders (huge size) as example:

117G hgpo.llo.prmt.processed-28 
117G hgpo.llo.prmt.processed-29 
117G hgpo.llo.prmt.processed-3 
117G hgpo.llo.prmt.processed-30 
117G hgpo.llo.prmt.processed-31 
117G hgpo.llo.prmt.processed-32

what is the best approach to delete the topic/s from the folder /var/kafka/kafka-logs ,

and what are the exactly steps to do so , as stop service before deletion etc .

second important question:

what is the mechanizem that suppose to delete automatically the topics ?

jango
  • 59
  • 2
  • 3
  • 12

1 Answers1

2

Use the appropriate log.retention.X broker properties, as described here. By default logs are retained for 168 hours (7 days) but you can set it as low as 1 millisecond (which is useful to completely clear a topic).

You can also set the retention period on a per-topic basis, using bin/kafka-topics.sh.

Before changing these properties, you should read and understand the log retention and compaction documentation.

Also be aware that the logs are not immediately purged, so if you continue to write to them you may run out of disk space before any old space is freed.

Lastly, check the Kafka broker runtime logs -- depending on what's happening on your brokers, they can also grow quite large. Use the truncate command on Linux to purge them.

guest
  • 36
  • 1