2

I was wondering if any of you encountered such an issue, that the reported load of the server is really high though none of the parameters seems to indicate that its busy at all. I would need to check this again, but I think this only happens after I start passenger. I am on a dedicated virtual host.

Any tips of why is this happening or ideas how to debug this load value? Here is my top reading, I can provide other information if needed.

top - 13:34:38 up 122 days, 19:15,  1 user,  load average: 2.44, 2.02, 1.85
Tasks:  51 total,   4 running,  47 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1048576k total,   797196k used,   251380k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached
dlackty
  • 103
  • 3
HouseMD
  • 41
  • 5
  • How many cores does the machine have? Use `grep -e "^processor" /proc/cpuinfo | wc -l` if you don't know. You need to take the number of cores in to account when looking at the load average. – Kenny Rasschaert Nov 08 '11 at 12:12
  • Do you have an issue? Sometimes load average numbers are high just because there's some processes waiting for other processes but all requests are served in timely manner. Also I think you should add swap so that the OS can put unused stuff out of the memory and use the memory for filesystem cache. – Antti Rytsölä Nov 08 '11 at 15:25

4 Answers4

2

When you see high load but low CPU usage it indicates that something else is the bottleneck. (As Janne said, a load average of 2 isn't particularly high but it's not a bad idea to understand what your system is doing in great depth.)

Disk access is the most common reason for high load averages with low CPU usage. You can see some stats about disk usage with vmstat, iostat or dstat depending on what you have installed.

For comparison, one of my fairly busy web servers is showing the bi column (block read in) in vmstat at 100 per 30 seconds and the bo column (blocks written out) as 3000 per 30 seconds. The load average on that server is about 1.

Another possible reason is a large number of processes that are doing very little work. Most servers I see have between 100 and 200 processes. If your server has 500 or 1000, that can cause a high load average.

Again, for comparison we recently had a large number of emails in our mail queue that caused around 1,000 processes to be spawned to send the emails. That caused the load average to go up to about 70 and made the system difficult to use.

Ladadadada
  • 26,337
  • 7
  • 59
  • 90
2

Thank You for Your input, as it turns out the right answer (and the key in my question) turned out to be that I am using a virtual dedicated server. The server had 0.6 of a processor unit (so 1.2Ghz out of 2Ghz Xeon). This caused weird readings in my top and server health monitoring software (new relic rpm called this time 'stolen' in cpu usage graph). I now updated to a full core VPS and the load is down to its proper value ~0.07.

HouseMD
  • 41
  • 5
0

Load average of 2 is really not high, actually it's low.

Are you seeing actual performance problems somewhere?

Janne Pikkarainen
  • 31,852
  • 4
  • 58
  • 81
  • 1
    Really? Somehow I figured it means "Server would have completed the whole load scheduled for the last second or so if he was 2 times as fast" as in [some load explanation](http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages) – HouseMD Nov 08 '11 at 12:10
  • Also I must stress the point that the server is 99.3% idle and I don't notice any performance issues, but I am wondering why this reading is displaying these values – HouseMD Nov 08 '11 at 12:11
  • You have found a very dramatic load average explanation from some random blog. In Linux "load" means your server is waiting for some resource (CPU, network, disk), but load averages around 2 _are_ low. At the moment my workstation is upgrading itself from Fedora 15 to Fedora 16, and is also running Windows 7 in VirtualBox. I also type in this reply to you in Firefox and have couple of other pieces of software running, too. Load average seems to hover around 4 and this workstation is still very responsible. – Janne Pikkarainen Nov 08 '11 at 12:23
  • Under some circumtances I've seen server loads to climb up to hundreds. At that point everything usually is very sluggish but the server still chucks along. In your case everything seems to be fine load and performance-wise. – Janne Pikkarainen Nov 08 '11 at 12:25
  • Hmm, my current machine has multiple IDE's running, browser with lots of tabs, a dev server of RoR, skype etc and its LOAD value is 0.5. Shouldn't a server with very low 1-2rpm web requests have a similar or even lower load value? And I wouldn't count the blog random [wikipedia](http://en.wikipedia.org/wiki/Load_%28computing%29) displays similar ideas. – HouseMD Nov 08 '11 at 13:05
  • In my workstation example the load was coming from multiple sources: hard drive was constantly accessed due the 1600 packages going to be upgraded and also yum using some CPU. My Firefox had as YouTube video running on one tab. Windows 7 in VirtualBox had Outlook 2010 doing something. That lead to ~4 load average but not any visible lag. My point is: especially in Linux load reading does not tell much. If some background process wants disk I/O but the process you need is CPU-bound, you wont see any lag. Don't panic. – Janne Pikkarainen Nov 08 '11 at 13:12
0

I usually see this when a process or two enter "D" state. This means they're waiting on I/O from something. It's possible that this could be an NFS share that's timing out or similar. If they are the only two processes waiting on this storage device, your %CPU in IOWAIT state won't go up much.

To be clear, a load average of 2.44 can be high or low depending on how many CPU threads your system has. If you have a dual-core Xeon with hyperthreading, you have 4 CPU threads. A load average of up to 4.0 would mean the system is at or under full capacity. A load average of over 4.0 on such a system would mean there are more processes in the run queue than there are CPU threads.

JakePaulus
  • 2,347
  • 16
  • 17