Load average 3.00 on mostly idle server

Question

I have a VPS hosting a small web app (apache, php-fpm, mysql) that's mostly idle for now as the traffic is quite low.

Surprisingly, the load average is always >= 3.00:

# uptime
 02:20:00 up 69 days,  6:03,  1 user,  load average: 3,04, 3,04, 3,05

I've been seeing this for weeks, and ended up considering this 3.00 as my 0.00. The CPU usage is close to 0%, the server is as fast as I would expect, and there is no other sign of server load.

But still, this stirs my curiosity. What can cause this permanent load average?

top shows an idle server:

top - 02:25:56 up 69 days,  6:09,  1 user,  load average: 3,07, 3,07, 3,05
Tasks: 218 total,   1 running, 216 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0,0 us,  0,3 sy,  0,0 ni, 99,7 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :  3878668 total,   165660 free,  1187816 used,  2525192 buff/cache
KiB Swap:  4079612 total,  3810976 free,   268636 used.  2132052 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2200 root      20   0       0      0      0 S   0,3  0,0   0:00.02 kworker/0:0
26243 mysql     20   0 3297012 702988   7740 S   0,3 18,1 248:38.28 mysqld
    1 root      20   0  188692   3456   2076 S   0,0  0,1  12:30.25 systemd
    2 root      20   0       0      0      0 S   0,0  0,0   0:03.11 kthreadd
    3 root      20   0       0      0      0 S   0,0  0,0   1:31.56 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0,0  0,0   0:00.00 kworker/0:+
    7 root      rt   0       0      0      0 S   0,0  0,0   0:01.09 migration/0
    8 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/0
   10 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/1
   11 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/2
   12 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/3
   13 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/4
   14 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/5
   15 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/6
   16 root      20   0       0      0      0 S   0,0  0,0   0:00.00 rcuob/7

I have read this question: CPU 100% idle but still showing load average but it didn't help.

Can this be a side effect of the server being virtualized?

Edit: iostat output:

# iostat -x 1 20
Linux 3.10.0-327.10.1.el7.x86_64 (centos02)     07/06/2016      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6,23    0,00    0,32    0,32    0,00   93,13

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,15    0,05    7,53     1,57    79,92    21,50     0,08   10,76   12,29   10,75   3,93   2,98
dm-0              0,00     0,00    0,00    0,04     0,01     0,14     8,01     0,03  842,38    6,66  908,04   0,35   0,00
dm-1              0,00     0,00    0,02    5,64     0,98    73,38    26,26     0,06   11,03   14,83   11,02   5,19   2,94
dm-2              0,00     0,00    0,03    0,09     0,57     6,39   112,18     0,05  370,36   10,75  482,51   4,09   0,05

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,03    0,00    1,52    0,00    0,00   95,45

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     1,00    0,00    9,00     0,00    64,00    14,22     0,05    5,33    0,00    5,33   5,11   4,60
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    8,00     0,00    64,00    16,00     0,05    6,12    0,00    6,12   5,75   4,60
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,50    0,00    0,50    1,01    0,00   97,99

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00   53,47     0,00   728,71    27,26     0,22    4,04    0,00    4,04   3,74  20,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00   39,60     0,00   728,71    36,80     0,22    5,45    0,00    5,45   5,05  20,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,51    0,00    0,00    0,00    0,00   99,49

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    6,00     0,00    16,00     5,33     0,03    4,17    0,00    4,17   4,17   2,50
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    4,00     0,00    16,00     8,00     0,03    6,25    0,00    6,25   6,25   2,50
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,50    0,00    0,00    0,00    0,00   99,50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00   16,00     0,00   104,00    13,00     0,13    8,25    0,00    8,25   0,62   1,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00   15,00     0,00   104,00    13,87     0,13    8,80    0,00    8,80   0,67   1,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,54    0,00    1,02    0,00    0,00   96,45

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00   29,00     0,00   736,00    50,76     0,28    9,69    0,00    9,69   2,28   6,60
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00   29,00     0,00   736,00    50,76     0,28    9,69    0,00    9,69   2,28   6,60
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,01    0,00    1,01    0,50    0,00   97,49

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    3,00     0,00    12,00     8,00     0,02    6,33    0,00    6,33   6,33   1,90
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    2,00     0,00    12,00    12,00     0,02    9,50    0,00    9,50   9,50   1,90
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,50    0,00    0,50    0,00    0,00   99,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    1,00     0,00     8,00    16,00     0,01    7,00    0,00    7,00   7,00   0,70
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    1,00     0,00     8,00    16,00     0,01    7,00    0,00    7,00   7,00   0,70
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,51    0,00    1,51    0,00    0,00   95,98

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2,02    0,00    0,00    0,00    0,00   97,98

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1,01    0,00    0,00    0,00    0,00   98,99

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     1,00    0,00   19,00     0,00   588,00    61,89     0,07    3,58    0,00    3,58   1,74   3,30
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00   20,00     0,00   588,00    58,80     0,07    3,45    0,00    3,45   1,65   3,30
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,00    0,00    0,00  100,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,00    0,00    0,50    0,00    0,00   99,50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,50    0,00    0,00    0,00    0,00   99,50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
dm-2              0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

Edit:

# ps -ALo stat,pid,comm | egrep '^(R|D)'

Dsl    857 prltoolsd
Dsl    857 prltoolsd
Dsl    857 prltoolsd
R+    2307 ps

Could we see some output from `iostat -x 1`? Not too many lines, and it can get overwhelming on systems with many physical devices attached, so I'm interested in devices with *consistently high* `%util` entries (say, over 50). — MadHatter, Jun 07 '16 at 10:32
Sure you can: http://pastebin.com/vHz42FQb As you can see, `%util` is close to `0` most of the time... — BenMorel, Jun 07 '16 at 13:20
Thanks for that. I've edited it into the question, as that helps questions stay relevant longer (pastebin output has been known to disappear). It doesn't look IO-related, at least not local IO. No NFS mounts? — MadHatter, Jun 07 '16 at 13:24
OK, thanks, it's not IO-blocking, then. I'll be interested to see if someone can chase it down. — MadHatter, Jun 07 '16 at 13:26
TBH this is a CentOS template created & installed by the local hosting company, and this would not be the first weirdness I've seen (odd default firewall config scripts, for example); I'm quite interested to find out the source of the load average as well! — BenMorel, Jun 07 '16 at 13:36
One more question, for which I apologise: are you **sure** this is a VPS? Not merely a containerised environment? — MadHatter, Jun 07 '16 at 13:52
Do you know what virtualization is being used ( openVZ/Xen/KVM?).. Do you know how the provider allocates the CPU? (EX: do you get a full core?, or do you get some GHZ out of a full CPU core?) — vijay rajah, Jun 07 '16 at 14:03
Also, Do you have "sysstat" (SAR) installed? If so, can you please go and check if the load avg. stays consistent or there is a wild variability in the averages — vijay rajah, Jun 07 '16 at 14:24
@vijayrajah I have no idea what virtualization is being used. I *think* when I have dedicated cores, but can't guarantee it. It looks like I have `sysstat` installed, but I do not know how to use it. Can you please give me the command to get the output you need? — BenMorel, Jun 07 '16 at 14:49
@MadHatter I'm not sure what you mean by containerized environment. I have root access on the machine. My hosting company calls it a VM. Could you please clarify? — BenMorel, Jun 07 '16 at 14:55
I don't know how to be clearer (though I freely acknowledge that companies offering containers as *virtualisation-lite* can be **very** evasive about them not being VMs). Having root access doesn't tell us whether you have a real virtualised environment, or a mere container. As Vijay asks, you may have to get your provider to clarify what technology they're using. Or try inserting a module into the kernel; the `usb_storage` module is pretty harmless, and isn't usually loaded on VMs (as they have no usb interfaces), so `modprobe usb_storage` should be fairly safe to run. — MadHatter, Jun 07 '16 at 15:32
@MadHatter I've followed [this answer](http://unix.stackexchange.com/a/89718/30018) on [Easy way to determine virtualization technology](http://unix.stackexchange.com/q/89714/30018), and `dmidecode -s system-product-name` tells me that I'm running on `Parallels Virtual Platform`. So it looks like I'm on a proper VM? — BenMorel, Jun 07 '16 at 16:18
Can you please share the output of `dmidecode -s system-product-name` This will enable us to identify the virtualization types (most of them atleast!)... please read up on `sar`it is a really usefull tool. please see [this](http://www.thegeekstuff.com/2011/03/sar-examples/) .. I'm intrested in `-q` , `-p` & `-d` for 24 hours. — vijay rajah, Jun 07 '16 at 16:22
Regarding `dmidecode`, I think our messages crossed, see my comment above, it's running Parallels. Regarding `sar`, sorry I don't have (yet) 24 hours -not sure why, as it was already installed- but here's the 4 hour output anyway: http://pastebin.com/Jjqpe0A1 — BenMorel, Jun 07 '16 at 17:53
Yes, it looks like it is a real VM, not just a container. Scratch that hypothesis as well. — MadHatter, Jun 08 '16 at 05:52
Do you have any (3) processes in uninterruptible sleep (D) ? - they would be counting towards your loadavg. — Kjetil Joergensen, Jun 09 '16 at 01:39
Please can you run `ps -ALo stat,pid,comm | egrep '^(R|D)'` and paste the output into your question? This should make it pretty obvious normally where the problem is. — Matthew Ife, Jun 09 '16 at 14:46
Loadavg 1 min, means the number of process with state READY waiting in the RUN QUEUE in the last minute. I'd `sudo perf top` or `strace -p [PID]` to see which functions/syscalls are taking over your processors, but I'd put my two cents on the `sleep` calls someone mentioned before. — Marcel, Jun 13 '16 at 09:11

score 6 · Accepted Answer · answered Jun 09 '16 at 21:46

6

From the output of ps -ALo stat,pid,comm | egrep '^(R|D)', it seems that you have three processes in uninterruptible sleep (D state). As Linux count processes in D state when calculating loadavg, maybe these three prltoolsd processes are the cause of the relatively high load average.

From a Google search, it seems that prltoolsd is part of the "Parallels Tools" suite, some sort of guest additions for the hosted image. As they are sleeping in kernel-level system calls (ie: uninterruptable sleep), maybe thery are listeing on a "fake" device to allow the host system to communicate with the guest system, hence the 3.0 load average value.

If you want to be really sure they are the cause of the problem, stop the prltoolsd processes and/or uninstall the tools. Anyway, I strongly suspect you can completely ignore them and the 3.0 load average...

answered Jun 09 '16 at 21:46

shodanshok

47,711
7
111
180

1

Looks like you found the cause of the problem, congrats! I'll check with the hosting company first, but I think I'll just ignore it. – BenMorel Jun 09 '16 at 22:30
I don't think so, actually it was me: http://serverfault.com/a/782429/67675 but you gave the output which didn't have any `D`s at that moment. ;) – poige Jun 12 '16 at 06:18
@poige Where should we have seen the `D`s? The load average is always the same, so I'm pretty sure it would have showed up. – BenMorel Jun 14 '16 at 08:42

Karen B · Answer 2 · 2016-06-02T06:17:58.573

4

First of all, load average is a useless metric without context, particularly knowing how many CPU cores your server has. A load average of 3 on a single-core server (somewhat busy) is very different from the same load average on one with, say, 64 cores.

That said, mysqld is your top process, and it's multi-threaded, meaning the one process can actually be using multiple CPU cores at once. Run ps -efT | grep mysqld to see how many active threads it has. top will also show threads with the -H option. Even if mysql isn't actively getting queries, it does run some background tasks. You may want to check its logs to see what it's doing.

edited Jun 02 '16 at 06:17

answered Jun 02 '16 at 06:10

Karen B

534
3
7

Good point. There are 2 CPU cores. About `mysqld`, it's here in this `top` output, but is not always active. Very often it's not even visible in the output. I cannot see anything in the logs that would explain any such activity from MySQL. – BenMorel Jun 02 '16 at 09:22
Ok, then, let's look at the memory usage. You have swap and it seems to be used. Swapping is obviously going to create I/O wait, which will increase the load average, especially if you have a full stack on the same host. If you haven't changed the kernel parameter `vm.swappiness` from the default of 60, you may want to change it to 1 (do not set it to 0 -- this disables swap): `sudo sysctl vm.swappiness=1`. If you can stop your apps temporarily, that would also be helpful in determining the instance's baseline load. This answer may also be helpful: http://unix.stackexchange.com/a/130970 – Karen B Jun 02 '16 at 21:33
Swap is almost unused, and there is plenty of RAM available (2GB out of 4GB), the memory "used" is for buffer/cache, it's therefore available. – BenMorel Jun 02 '16 at 21:37
The `vmstat` output pointed to below poige's answer is suggestive that it's not swapping. I agree it's a very short dataset, but if the box was continuously swapping you'd see it even in such a short snippet. – MadHatter Jun 07 '16 at 13:54

poige · Answer 3 · 2016-06-07T14:46:02.223

2

On Linux a process that got stuck in I/O would also be counted as placed into run-queue. This is a ~~bug~~ feature. You can check/show vmstat 1 2's output, to see if you have any blocked, for e. g..

edited Jun 07 '16 at 14:46

answered Jun 07 '16 at 10:05

poige

9,448
2
25
52

Here is the `vmstat` output: http://pastebin.com/qtnAqJxB – BenMorel Jun 07 '16 at 13:23
my bad. `vmstat 1 2`, please – poige Jun 07 '16 at 14:45
Here: http://pastebin.com/SRUs9Fyq – BenMorel Jun 07 '16 at 16:07
ok, it doesn't look like anything blocked on I/O. Next: `ps -A -o s | uniq -c | sort -n` ? – poige Jun 07 '16 at 18:12
+ `cat /proc/loadavg` – poige Jun 07 '16 at 18:30
Here: http://pastebin.com/ZeDvHusv – BenMorel Jun 07 '16 at 21:33
ok, can I have ssh access there? O_o (LOL) – poige Jun 08 '16 at 02:20
I'm afraid not :) – BenMorel Jun 08 '16 at 10:25
You're in danger otherwise anyways. ;-P – poige Jun 08 '16 at 10:45

score 0 · Answer 4 · answered Jun 09 '16 at 18:59

0

i would check connections to my server (/var/log/secure and /usr/local/apache/logs/access_log) a useful way to find out ip's that are making multiple connections to server

netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Also, check if the node that's holding that vm isn't under high load or is having issues with disk(s).

answered Jun 09 '16 at 18:59

RonanW.

429
2
7

Your command returns less than 10 IP addresses, so I don't think this is an issue; and if many clients were connected and created some load, wouldn't that be visible in the commands others have suggested? I've opened a ticket with the hosting company, we'll see if it's an hypervisor problem. – BenMorel Jun 09 '16 at 21:28
When you don't see processes that are creating load, abusive network behaviour can be one of reasons. In that case, you wouldn't see processes with high cpu/memory usage, but you could catch abuse by looking at those logs. Also, run iptraf and look at traffic on public interface (look for udp connections) – RonanW. Jun 10 '16 at 17:28

Load average 3.00 on mostly idle server

4 Answers4