4

I'm having problems with a FreeBSD 9 server. One of the 8 cores (4 w/ HT) is constantly at 100% kernel load.

top gives me:

CPU:  0.0% user,  0.0% nice, 12.5% system,  0.0% interrupt, 87.5% idle

Notice that 12.5% is exactly 100%/8.

top -S shows me that geom is the problem:

13 root        3  -8    -     0K    48K -       7 385.2H 100.00% geom

I checked with iostat if there's something IO-heavy going on, but the server is currently idle:

pascal@srv ~$ iostat 1 5  
       tty            ada0             ada1            pass1             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0     6 22.07   6  0.13  22.43   9  0.20   0.00   0  0.00   0  0  4  0 96
   0   234  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0 13  0 87
   0    78  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0 12  0 87
   0    78  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0 13  0 87
   0    78  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0 12  0 88

There's also plenty of free RAM

Mem: 1174M Active, 4399M Inact, 8241M Wired, 208M Cache, 1726M Free

There's no swap configured, but I noticed some strange errors in dmesg.today

swap_pager: I/O error - pagein failed; blkno 288,size 12288, error 6
vm_fault: pager read error, pid 95209 (nginx)

There are about 30 of these per day. I added a swap file but they persist. I also stopped nginx, but the load doesn't go down.

Where can I find more information on why geom is using so much CPU? Thanks in advance.

  • Did you have a swap file before? It looks like it's constantly virtual memory page faulting, but you obviously have free memory. Strange. – EhevuTov Nov 13 '12 at 05:58
  • No, I haven't had a swap file before. The server has 16GB of RAM and roughly 1.5GB is used. Also, after I added the swap file, these errors kept appearing. `Swap: 4160M Total, 836K Used, 4159M Free` –  Nov 13 '12 at 08:48
  • I'm adding this as a comment here as this is no real answer. The problem was a broken harddrive. 15 hours after I asked this question ada0 simply disappeared in /dev. The hdd was replaced, I rebooted and the load is gone. –  Nov 13 '12 at 14:11
  • oh, awesome. So it was constantly rebuilding or something. Thank you for the feedback. – EhevuTov Nov 13 '12 at 20:39

1 Answers1

2

Think this was really this issue in FreeBSD,

Bug 171865

You resolved it by rebooting.

CharlesR
  • 36
  • 2
  • Thanks. The bug report also shows how the reporter found the cause of the high load: `sudo dtrace -n 'fbt:kernel:g_*:entry { @[probefunc, stack()] = count(); } tick-1sec { trunc(@, 3); printa(@); ` –  Dec 13 '13 at 11:09