2

I've been experiencing a spike in load average on a web server I manage on almost a daily basis now, here are the server specs:

  • 6 x 2.4 GHz dedicated CPU
  • 3GB RAM

This is a VPS of which is running debian 6, I installed apache, php and mysql via apt. I'm not sure if there is a configuration that I've gotten wrong.

Today the load average peaked so high the server failed to serve the web application (WordPress). The screenshot below shows our server monitoring system. You'll notice the high load average correlates to a high apache busy worker count, and subsequently the memory maxes out too.

enter image description here

After forcing a reboot on the server I still have a higher than usual load average, despite the CPU usage being low. The following screenshots show htop and then iotop.

enter image description here

enter image description here

The load average is now > 6, here's what the apache server status says:

enter image description here

I'm really struggling with how to investigate this. Can anyone assist in figuring this one out.

Update 1

I've search the apache error logs and no word of anything hitting max execution time. I do, however, get a lot of the following... starting just as the server started to load up:

::1 - - [24/Feb/2014:15:03:31 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:32 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:33 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:34 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:35 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:36 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:37 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:38 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:39 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"
::1 - - [24/Feb/2014:15:03:41 +0000] "OPTIONS * HTTP/1.0" 200 152 "-" "Apache/2.2.16 (Debian) (internal dummy connection)"

Note how they're all about 1 second after the previous... perhaps this is something.

Update 2

So I had the server host move the VPS to a new hypervisor, however afterwards it's still has quite a high iowait. I ran iostat 1 and this is what I recieved:

enter image description here

Does this help identify the problem?

Ben Everard
  • 599
  • 3
  • 7
  • 22
  • Consider using `iotop` tool to see if there's some unaccounted disk activity going on. Also consider using some monitoring tool that can show `cpu steal` time. If your host is severely over-provisioned, that might indicate why load is high but no processes uses it. – Mxx Feb 24 '14 at 16:32
  • The server just peaked again... I quickly showed `sudo iotop` and apache was writing a lot to the server, I'm not sure where I can find logs to say what what being written and at what time... can you advise? – Ben Everard Feb 25 '14 at 14:07
  • check in `/var/log/apache/` to see if you have large log files and perhaps if they are being rotated right now. – Mxx Feb 25 '14 at 14:11
  • Here is a screenshot of the apache log dir, sorted by size --> http://cl.ly/image/3Z0r2c1w0c2R. Here I caught apache reading lots from the disk --> http://cl.ly/image/3b3I3s2y3y2K. – Ben Everard Feb 25 '14 at 14:26

2 Answers2

1

It looks like you've got a script somewhere that is causing the load.

Start by going through your apache error log and looking for max_execution times or timeouts. Move on to the access logs and look for scripts that are being accessed which may be causing the hanging.

Daniel Widrick
  • 3,488
  • 2
  • 13
  • 27
  • Hi there, I used `grep` to find anything hitting the max_execution, but nothing came back. I did find, however, lots of strange entries in the access log. See "Update 1" in my question. Thanks. – Ben Everard Feb 24 '14 at 16:13
  • Go ahead and google that message but essentially: you can ignore it. It should not be the issue unless something is horribly misconfigured. – Daniel Widrick Feb 24 '14 at 16:17
  • Yeah a quick Google suggested it was just a request from the server to itself... just seemed funny it happened at the same time. – Ben Everard Feb 24 '14 at 16:18
0

There are a few things you can do to investigate the problem, including using vmstat 2 (for example - this will display the output of key resources every 2 seconds).

One thing which jumps out at me though is the amount of swap being used - 841MB on a server with 3 gigs is very substantial. I suspect that your system is swapping, causing IO to go very high and pushing up the load etc. If this hypothesis is correct, the solution is to deal with the swapping.

You either need to throw more memory at the system or change how swap is handled or both. I'd suggest starting with the latter - its easy to do just configure swappiness. To do this type echo 'vm.swappiness = 10' >> /etc/sysctl.conf and then "sysctl -p". This will make the CPU do more work but swap less. On many VM's, disk IO is a bottleneck, so the affects are pretty instant and remarkable.

Throwing more RAM will also reduce the amount of swapping and speed up the system.

davidgo
  • 6,222
  • 3
  • 23
  • 41