Unusual load average for an idle workstation

Question

I have a new workstation with an unusual high load average. The machine specs are:

Xeon CPUs
256 GB of RAM
4 x 512 GB SSD connected to a LSI2108 RAID controller (in raid 5)

I have installed CentOS 6.5 64 bit from the live cd, configured the partitions (using LVM), the network, the users/groups, and installed some software like development tools and MATLAB. The workstation load average, after few minutes from the boot, has a value ranging from 0.5 to 0.9. But it is not doing anything. Thus I cannot understand why the load average is so high.

Could you help me to diagnose this issue?

Here are some information:

uname -a

Linux delta.grid 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

top

top - 15:44:53 up 22 min,  1 user,  load average: 0.84, 0.69, 0.39
Tasks: 729 total,   1 running, 728 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  264422908k total,  1986152k used, 262436756k free,    25640k buffers
Swap: 134217720k total,        0k used, 134217720k free,   147784k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                        
 4094 root      20   0 15564 1764  944 R  0.3  0.0   0:01.44 top                                                            
    1 root      20   0 19364 1536 1228 S  0.0  0.0   0:01.26 init                                                           
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd                                                       
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.27 migration/0                                                    
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0                                                    
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                    
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                     
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.14 migration/1                                                    
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                    
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1                                                    
   10 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/1                                                     
   11 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                                    
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                                    
   13 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/2                                                    
   14 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/2                                                     
   15 root      RT   0     0    0    0 S  0.0  0.0   0:00.03 migration/3                                                    
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3                                                    
   17 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/3                                                    
   18 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/3                                                     
   19 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/4                                                    
   20 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/4                                                    
   21 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/4                                                    
   22 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/4                                                     
   23 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/5                                                    
   24 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/5                                                    
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/5                                                    
   26 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/5                                                     
   27 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/6                                                    
   28 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/6                                                    
   29 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/6                                                    
   30 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/6                                                     
   31 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/7                                                    
   32 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/7

free -m

             total       used       free     shared    buffers     cached
Mem:        258225       1939     256286          0         25        144
-/+ buffers/cache:       1769     256455
Swap:       131071          0     131071

iotop

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                        
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    4 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
   10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
   14 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   15 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
   18 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   19 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   20 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
   22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
   24 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
   25 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   26 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]
   27 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/6]
   28 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/6]
   29 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/6]
   30 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/6]
   31 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/7]
   32 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/7]
   33 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/7]
   34 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/7]
   35 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/8]
   36 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/8]
   37 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/8]
   38 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/8]
   39 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/9]
   40 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/9]
   41 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/9]
   42 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/9]
   43 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/10]
   44 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/10]
   45 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/10]
   46 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/10]
   47 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/11]
   48 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/11]
   49 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/11]
   50 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/11]
   51 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/12]
   52 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/12]
   53 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/12]

vmstat 5 5

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 262437312  25728 147808    0    0     3     0    4    5  0  0 100  0  0    
 0  0      0 262437296  25728 147808    0    0     0     0   58  101  0  0 100  0  0    
 1  0      0 262437312  25728 147808    0    0     0     0   64  117  0  0 100  0  0    
 2  0      0 262437312  25728 147808    0    0     0     0   48   95  0  0 100  0  0    
 1  0      0 262437200  25728 147808    0    0     0     0   58  117  0  0 100  0  0

I have not enough space to copy/paste the output of 'ps axf', but: - there are no processes in D state (disk sleep - uninterruptible) - all the processes are in S state except the monitoring tool executed (like top or iotop)

Thanks in advance for your help!

What about if you wait for 15 -30 minutes after the boot? Paste top and vmstat after that. — Danila Ladner, Dec 13 '13 at 16:41
It does not change. It starts the unusual load average 5/10 minutes after the boot, and it continues forever. For example, after an "idle weekend" the current load average values are: 1.03 0.94 0.73. — marco skv, Dec 16 '13 at 08:03
Try to monitor `/proc/interrupts`. Also You can try disabling/disconnecting network interfaces (I've seen it as a source of load in the past). — kestasx, Dec 31 '14 at 23:24

score 2 · Answer 1 · answered Aug 27 '15 at 15:03

2

This bug is documented upstream in https://access.redhat.com/solutions/508303 and so far the solution indeed is to disable the acpi_pad driver.

On my systems (IBM Flex x240) this happens after disabling Hyperthreading.

answered Aug 27 '15 at 15:03

Marco d'Itri

213
1
6

score 1 · Answer 2 · answered Dec 31 '14 at 22:56

Without having more details about your server, I am just going to add something I have run into on a handful of servers with Xeon procs. There is a known ACPI bug that was "fixed" but not entirely fixed that affects CentOS 6 servers and some hardware.

You might try adding the following to your kernel boot commands to see if it helps (requires reboot):

grubby --update-kernel=ALL --args="acpi_pad.disable=1"

If that does not help, be sure to remove it:

grubby --update-kernel=ALL --remove-args="acpi_pad.disable"

I assume you have also looked through dmesg for any devices that could not be initialized. Some devices are supposed to be blacklisted by /etc/modprobe.d/blacklist.conf via the "hwdata" RPM.

You may also want to enable thread view in "top". There are additional tools that help debugging kernel threads that are waiting, but that is a much bigger topic.

Hi Aaron, and thanks for the suggestion. Unfortunately I have changed my job and I cannot access anymore that workstation. But I forwarded the suggestion to my former colleague. If he manages to fix the problem, I will let you know ;) — marco skv, Jan 06 '15 at 20:55

Unusual load average for an idle workstation

2 Answers2