Strange behavior with `find -delete`

Question

I'm using Ubuntu 12.04 with 2 x 2TB SATA3 harddisks in software RAID1. The filesystem is ext4 with data=writeback.

When using find . -maxdepth 1 -name '*' -delete to delete a large number of files in a directory, doing df at intervals show that:

Initially a large number of files get deleted, and iotop shows the find operation taking most of the IO.

enter image description here

Later df shows that the number of files remain constant, and iotop shows that IO usage is now dominated by jbd2 and md4_raid1.

enter image description here

Few minutes later, df shows the number of files dropping once more.

Can anyone explain this behavior? Is this normal or is there something wrong with my server? Also, how can 3 processes all have 99.99% IO usage at the same time?

score 2 · Answer 1 · answered Mar 14 '13 at 13:39

2

It's what I would expect. Find finds files - i.e. it reads from the VFS, when the data is not in cache it needs to be fetched from disk. As it processes each file, it enqueues the changes to be written. At some point the write buffer gets full and the changes must be written to the disk - at this point the process doing the writes will be blocked making further write OR READ requests until there's space to buffer more writes.

answered Mar 14 '13 at 13:39

symcbean

21,009
1
31
52

When this mass deleting is done, the server slows to an almost complete halt. Is there a way for the server to avoid getting thrashed by the write block? – Nyxynyx Mar 14 '13 at 13:43
Don't delete so many files all at once? – Michael Hampton Mar 14 '13 at 13:50
See http://serverfault.com/questions/487599/running-a-process-within-a-limited-amount-of-resources-in-linux – symcbean Mar 14 '13 at 14:25

score 0 · Answer 2 · answered Mar 15 '13 at 15:55

0

You can prevent the system from being blocked by reducing the I/O priority of the process:

ionice -c3 nice -15 find ... -delete

If that's not enough then you can use chrt to put the process into the SCHED_IDLE class.

answered Mar 15 '13 at 15:55

Hauke Laging

5,285
2
24
40

Should I actually reduce the IO priority of `[jbd2/*]` and `[md4_raid1]`? Will that slow down my entire IO subsystem even more? – Nyxynyx Mar 15 '13 at 17:00
Don't touch that kernel stuff. I even doubt that it's possible to change their priority anyway. – Hauke Laging Mar 15 '13 at 17:05

Strange behavior with `find -delete`

2 Answers2