delete millions of folders and files efficiently

Question

I have a server with a huge file system that is used for continues writes and reads without hiccups and needs to be active at all times.

My goal is to have a cronjob that deletes empty folders and files based on name and older than X amount of days. The obvious answer is to use find -name -mtime -delete but it uses too much disk resources and causing IO wait and other programs to miss their checks because of server unresponsiveness.

Is there any easy to use tool that does this kind of work? maybe limit disk usage? something else i haven't thought of?

Maybe [this question](https://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit) points in the right direction? — bjoster, May 07 '19 at 09:07

score 5 · Answer 1 · answered May 07 '19 at 08:47

Use ionice to reduce the priority of your find command and minimize the impact on other applications.

ionice -c 3 find -name -mtime -delete

-c 3 is class "idle" and a program running with idle io priority will only get disk time when no other program has asked for disk IO (for a defined grace period). The impact of idle IO processes on normal system activity should be zero.

If that never happens (likely on a system that sees use 24x7) try for instance -c 2 -n 7 for the lowest best-effort priority

score 2 · Answer 2 · answered May 08 '19 at 12:52

Most file systems do not deal with changes to millions of files efficiently, they do a tremendous amount of metadata I/O. Hopefully you have less than 100,000 files per directory.

Throttling with io priorities and/or cgroups can make the system responsive, but may cause this cleanup to not complete in a reasonable time.

On Linux LVM, you can add a caching layer with lvmcache. Add the fastest SSD you have so it can service several thousand IOPS. Throttling is probably still needed, if you require your I/O response times to not degrade.

delete millions of folders and files efficiently

2 Answers2