Why is DU causing high CPU usage?

Question

On CentOS 6.5, du -sk /home and sh -c (du -sk \/home) 2>/dev/null run because a scheduled disk usage for /home, but du is causing 100% CPU usage. Plus, before it's even finished, another du gets run and it piles up, bringing the system down to its knees.

I've since disabled the automated checks, but I can't run du -sk /home manually either. du runs fine (fast) on other directories (though /home is the largest - 4912620 - it shouldn't take as long as it does)

I would like to find out why is DU causing high CPU usage on /home

UPDATE 1: A full restart (host+guest) solved the problem.

UPDATE 2: Problem came back. While du is running, CPU usage is split nearly half and half between (us) and (sy). (wa) never creeps over 1%. Also, IOWAIT shows only disk writes?

It's likely to be iowait. Try running top or vmstat or iotop while doing du. — , Dec 28 '13 at 17:32
@EricDANNIELOU: surely IO wait would cause the du to relinquish the cpu ? — user9517, Dec 28 '13 at 17:36
@lain I'm not sure about what you mean. 100% cpu can be caused by iowait (wa), userland (us) or kernel (sy). — , Dec 28 '13 at 17:40
Iowait can definitely cause the CPU load to creep up. If the disks are slow, particularly network disks, then this is more common. — jaseeey, Jan 07 '14 at 11:45
Disk was an LVM volume for the guest KVM OS. OpCenter crew ran tests and found no physical problems with the disk subsystem. — Gaia, Jan 07 '14 at 19:42

score 1 · Answer 1 · answered Dec 28 '13 at 15:55

1

I can't tell you why exactly, but here's what I'd do to narrow down what is going on:

cd /home
for i in * ; do echo "Starting in ${i}..." ; du -s ${i} ; done

This will run "du -s" for each command inside the /home directory, and show you what it is about to do. This is also assuming you are using sh or bash as your shell.

When you find which directory is causing the problem, move in one directory more, and see what happens there. Also remember this will not check for hidden ("dot") directories at the level you start the for loop from. You may need to check those as well, or modify the glob expression to include them.

answered Dec 28 '13 at 15:55

Michael Graff

6,668
1
24
36

There seem to be no abnormal delays or CPU spikes when I run the above command. If I run it for only /home with `--max-depth=1` I also don't get a delay/spike. It's only when I run `du -sk /home`. – Gaia Jan 26 '14 at 16:43
Do you have any .dot directories in /home? – Michael Graff Jan 27 '14 at 21:24
Directories named ".dot" ? – Gaia Jan 28 '14 at 17:48
No, hidden files or directories inside /home – Michael Graff Jan 28 '14 at 17:54
yes, .htaccess, etc – Gaia Jan 28 '14 at 19:20
Is it possible one of those is the culprit? Try: ````du -sh .??*```` from ````home```` and see what happens. – Michael Graff Jan 28 '14 at 19:20
du: cannot access `.??*': No such file or directory – Gaia Jan 28 '14 at 19:22
You ran that from ````/home```` ? If so, then you don't have any hidden directories inside /home itself... – Michael Graff Jan 28 '14 at 19:22
yes. shall we continue in chat? – Gaia Jan 28 '14 at 19:23
let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/12715/discussion-between-gaia-and-michael-graff) – Gaia Jan 28 '14 at 19:25

Gaia · Accepted Answer · 2014-04-13T02:33:47.600

0

I finally discovered what the problem was when I came across this post.

root> modprobe vhost_net root> echo vhost_net > /etc/modules

For some reason, the virtio interfaces were not loading the driver before. All is good now.

edited Apr 13 '14 at 02:33

answered Jan 30 '14 at 04:56

Gaia

1,855
5
34
60

Why is DU causing high CPU usage?

2 Answers2