3

One of our servers is currently sitting with its load averages in the billions (with a b), while reporting that it only has 1,031 resident threads:

top - 12:04:26 up 2 days, 19:03, 68 users,  load average: 381713318.29, 2612390757.45, 2824329668.69
Threads: 1031 total,  10 running, 1019 sleeping,   2 stopped,   0 zombie
%Cpu(s): 34.0 us,  6.8 sy,  0.0 ni, 38.5 id, 20.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 49284928 total,  3913012 free,  3702836 used, 41669080 buff/cache
KiB Swap: 12582908 total, 12575140 free,     7768 used. 43752932 avail Mem

Other than the bizarrely high reported load averages, the machine seems to be working completely normally. It is responsive, there is nothing nothing unusual in syslog, etc. This is a CentOS 7 machine running kernel 3.10.0-327.36.3.el7.x86_64.

Have any of you seen something like this before, and/or can anyone explain what it would even mean for a machine to have a higher load average than thread count?

Juffo-Wup
  • 39
  • 1
  • Is it a VM? vSphere? How many CPU cores? How many disks? output of `ps axf`? `ps axm`? With 20% wait time I suspect something around I/O is inflating the calculation. – MikeyB Mar 24 '17 at 14:59
  • This is a physical (bare metal) machine with 12 physical CPU cores exposing 12 threads to the OS (that is to say, hyperthreading is disabled). The machine has one local hard drive and mounts a remote Lustre filesystem and several NFS filesystems. I can't paste the entire ps axf and ps axm outputs due to user privacy reasons, but the outputs look normal enough to me; ps axf gives 614 lines of output and ps axm gives 1,356 lines. There are four user processes in the 'D' state, but each of those processes only has one thread; I would think that could only increase the load averages by 4. – Juffo-Wup Mar 24 '17 at 15:18

1 Answers1

2

My best guess is a bug in the lustre code that is artificially inflating the load average.

Do you have monitoring on the server such that you can observe how the load average is changing over time? Is it monotonically increasing? Geometric? Or does it increase to a huge number immediately on boot/filesystem mount/user login and stay there?

MikeyB
  • 39,291
  • 10
  • 105
  • 189
  • We don't currently monitor load average over time, but will be starting to soon. So far this is a one-time problem; we are draining the machine of jobs and users and will reboot it. After that, it's anyone's guess as to whether the problem returns. – Juffo-Wup Mar 29 '17 at 19:15