How high can system load go?

Question

Every time I login to my servers via SSH it flashes me a bit of server info. IE IP Address, Swap Usage, Memory Usage, etc. One of the other things it flashes me is System Load. Now most of the time the system load number is < 0.10 but other times it I've seen it go up to 0.89 (usually around boot).

This this brings up the question, how high can system load go? For instance is it possible for it to go up to 2.00 even 100.00?

The fractional part indicates the CPU usage, the decimal part the number of waiting processes. Something like `100.89` would be more likely than `100.00` — Lekensteyn, Jun 28 '11 at 14:56
possible duplicate of [Acceptable load average](http://serverfault.com/questions/63319/acceptable-load-average) — Caleb, Jun 28 '11 at 14:59
@Caleb, I disagree that it's a duplicate. It's possible that the OP *intended* to ask something similar, like "how high can system load go (and still run acceptably)?". But as phrased, it's not the same question. — Cyclops, Jun 28 '11 at 18:10
@Cyclops: If the OP did not intend to ask about acceptable levels on servers and was asking a purely theoretical question about how *nix systems worked wouldn't that be off topic here and a candidate for migrating to unix.SE? — Caleb, Jun 28 '11 at 21:28

score 17 · Accepted Answer · answered Jun 28 '11 at 11:15

This this brings up the question, how high can system load go? For instance is it possible for it to go up to 2.00 even 100.00?

Absolutely. Looking at the uptime man page:

   System load averages is the average number of processes that are either
   in a runnable or uninterruptable state.  A process in a runnable  state
   is  either  using the CPU or waiting to use the CPU. A process in unin‐
   terruptable state is waiting for some I/O access, eg waiting for  disk.
   The  averages  are  taken over the three time intervals.  Load averages
   are not normalized for the number of CPUs in a system, so a load  aver‐
   age  of 1 means a single CPU system is loaded all the time while on a 4
   CPU system it means it was idle 75% of the time.

So if you have a lot of processes waiting to run (or a lot of processes blocked waiting for I/O), you're going to have a high load average. This article talks about it in more detail, and has useful links to other resources.

On an unloaded system, the load average will typically be in the range 0 <= load_average <= n, where n is the number of cores on your system.

Yep, Waiting for I/O is an easy way to create a lot of load: Mount an nfs share, turn of the nfs server, start up 10.000 processes that try to touch something on the nfs share: boom, load of 10.000 — Jens Timmerman, Nov 29 '12 at 08:36

Caleb · Answer 2 · 2011-06-28T14:32:26.830

5

I've seen alive systems hit the thousands. Load Average a relative measure based on waiting processes of how much competition there is for getting the kernels attention and being granted some time on the CPU. If the machine is swamped with jobs or crashing, that can take a long time.

What level is acceptable depends on the machine, the number of cores, the kind of kernel job scheduler in use, and the jobs you expect it to do. I have some machines that are quite happy in the ~10 range but bog down if they hit ~40-50. Others become noticeably laggy at 2 and would be unusable at 10.

It's not unusual for the load to be high durring boot since lots of things are being done at once and the machine is winding up. I would consider ~1 quite a normal load to hit durring boot for a desktop Linux, then settling down to ~0.1 while doing nothing.

edited Jun 28 '11 at 14:32

answered Jun 28 '11 at 11:08

Caleb

11,813
4
36
49

3

Minor correction: Load Average is not a measure of time. It's the average number of processes waiting. There's no way to tell the difference between 1) one process hogging a CPU and another one never getting it, and 2) two processes continuously trading off. They'd both be loads of 1.0. – Plutor Jun 28 '11 at 14:24
1

I've seen 1500+ on a seriously fubared linux box. Then it died. – Tom O'Connor Jun 28 '11 at 21:32

score 3 · Answer 3 · answered Jun 28 '11 at 11:23

On Linux, the system load average values are comprised of processes in one of three different states. In general, one could say that the load average is the amount of processes waiting for CPU time or consuming CPU time. The three values in the load average overview are the load average over the past minute, the last 5 minutes and the last 15 minutes.

The three different states of processes counted towards the load average are: (1) processes running on the CPU, (2) processes waiting for CPU time and (3) processes in uninterruptable sleep. The last category, while not generating CPU load, can increase the system load average significantly.

For example, a dozen processes waiting for reads from a disk that is very busy or unavailable will generate a load average of 12 as processes in uninterruptable sleep, but your CPU can be perfectly idle in the meantime.

So, yes, load average can easily go up to double digits. How bad that is is rather dependent on your hardware. If you have 16 cores, having 16 processes waiting for CPU time is not so bad. On a single core machine, having 3 processes waiting for CPU time can be very bad.

I think it is bad only if they are CPU-bound processes, which is rare. Most big processes are IO-bound or/and RAM-bound. — peterh, Apr 01 '20 at 10:41

score 3 · Answer 4 · answered Jan 10 '17 at 17:35

Make a simple C process running infinite loops in 10000 threads. Give it a very low priority (+20). Your load will be 10000, while your system will be still usable. It will use only a very few RAM (at most some megabytes).

Although it is a quite uncommon configuration, in real systems you won't find this.

The system load means only the mean number of processes waiting for a cpu time slot, no less and no more. Here is another answer about the proper way to interpret system load.

In the daily experience, a load over 30+ means mostly some problem.

score 2 · Answer 5 · answered Jun 28 '11 at 11:18

2

A few seconds after killing a process that was eating an old 450Mhz cpu:

enter image description here

answered Jun 28 '11 at 11:18

petrus

5,297
26
42

score 1 · Answer 6 · edited Jan 10 '17 at 18:03

1

I've seen a server running at >200 of load.

I would stress test and see.

edited Jan 10 '17 at 18:03

answered Jun 28 '11 at 11:07

Kedare

1,786
4
20
37

score 1 · Answer 7 · answered Jun 28 '11 at 20:38

When you repeatedly start processes that block immediately, the load will be as high as the number of processes you manage to start. Assuming you have enough RAM that the system is not swapping, the system will even stay responsive as load due to blocking processes is not really harmful.

How high can system load go?

7 Answers7

Linked