0

I recently moved my site from a VPS to a dedicated server. The VPS was around the same speed/spec with the dedicated one. It's usual load average was around .7-.9 (it was a supposed 3 core system).

Now I moved to a dedicated server with a dual core Core(TM)2 Duo CPU E7500 @ 2.93GHz. On it's lows, minute load average is around the same levels of the VPS.

The application serves around 200000 pages per day, each page is rendered in under 10ms and it contains some small images (~20 images of less than 5kb each).

The only hog that I have is a page with a cURL request of user's sites that have a timeout limit of 7 seconds and this page is called around 6000 times per day (usually the calls are grouped in time).

I feel that this is the culprit that is producing the loadavg spikes but I am not sure how to setup my configuration so as to avoid these nerve-racking spikes of satan that were non-existent in the VPS!

BTW the setup is fcgi on a typical cPanel apache config.

Any pointers on where to begin configuring/debugging would be most appreciated!

Thanks!!!

This is the output from top on a typical spike that I get. You see that CPU is lazily wondering about and if i "cat /proc/cpuinfo" it is @1600MHz!!!! (it is throttling)

-----------------------------------------------------------------------------
top - 17:02:39 up 16 days, 15:20,  1 user,  load average: 6.39, 2.28, 1.57
Tasks: 161 total,   1 running, 160 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.4%us,  1.8%sy,  0.0%ni, 81.8%id,  3.3%wa,  0.2%hi,  0.5%si,  0.0%st
Mem:   4046576k total,  4012780k used,    33796k free,   272176k buffers
Swap:  2104504k total,      104k used,  2104400k free,  2497684k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16800 nobody    18   0  422m  13m 2148 S  2.7  0.3   0:16.73 httpd
23746 nobody    18   0  419m 9796 2132 S  2.7  0.2   0:01.54 httpd
 2152 nobody    18   0  422m  22m  10m S  2.3  0.6   0:51.42 httpd
 2109 nobody    18   0  423m  16m 2872 S  2.0  0.4   0:49.05 httpd
 2113 nobody    18   0  422m  14m 2268 S  2.0  0.4   0:47.32 httpd
 2160 nobody    18   0  423m  14m 2252 S  2.0  0.4   0:50.83 httpd
 2195 nobody    18   0  487m  15m 2256 S  2.0  0.4   0:50.64 httpd
 8452 nobody    18   0  421m  22m  10m S  1.7  0.6   0:38.82 httpd
18044 sitenam   15   0  137m  16m 8128 S  1.7  0.4   0:02.93 php
 2150 nobody    18   0  422m  14m 2232 S  1.3  0.4   0:48.56 httpd
 2154 nobody    18   0  422m  22m  10m S  1.3  0.6   0:47.79 httpd
 2156 nobody    18   0  423m  23m  10m S  1.3  0.6   0:51.32 httpd
 2158 nobody    18   0  422m  22m  10m S  1.3  0.6   0:51.88 httpd
23919 nobody    18   0  417m 8368 2128 S  1.3  0.2   0:01.13 httpd
18045 sitenam   16   0  139m  18m 8068 S  1.0  0.5   0:03.02 php
 2111 nobody    18   0  549m  14m 2176 S  0.7  0.4   0:51.74 httpd
 5097 mysql     15   0  280m  34m 4596 S  0.7  0.9  13:02.88 mysqld
16981 sitenam   16   0  137m  16m 8080 S  0.7  0.4   0:03.01 php
18043 sitenam   16   0  138m  17m 8064 S  0.7  0.4   0:01.61 php
 2102 nobody    18   0  438m  33m  10m S  0.3  0.8   0:49.20 httpd
19566 sitenam   15   0 12744 1132  808 R  0.3  0.0   0:01.67 top
23800 nobody    18   0  418m 9164 2124 S  0.3  0.2   0:01.72 httpd
    1 root      15   0 10352  696  588 S  0.0  0.0   0:06.01 init
pataroulis
  • 143
  • 1
  • 2
  • 11

2 Answers2

0

If you have not done this turn off keepalive in apache. Keepalive will keep the connection open for the given seconds in the config. If you are serving a lot of pages with images it will keep connections open.

For example some browsers will open 4-6 connections at the same time to the webserver. So if so each client at any given time will have 4-6 connections open and if your keepalive timeout is 120 (i think that is the default) and in that 2 minutes you have 200 clients.. Your server might be in trouble.

So it turn it off and restart apache

KeepAlive Off

I think it is this due to your top.. nothing is really using that much cpu. So try that first.

top is not just a measure of cpu. It is also a measure of disk I/O and RAM to name two more.

Mike
  • 22,310
  • 7
  • 56
  • 79
  • Hi Mike! Thanks for your answer! I tried setting keepAlive to off, rebuild the configuration/restarted the server but the spikes were there. I was thinking of searching for bottlenecks elsewhere and drop the theory that the cURL process that takes 5000ms produces the spikes. One thing that I didn't mention is that the Dedicated server has a simple 7200rpm sata drive. Also, I was thinking of configuring apache as a mpm-worker instead of prefork. I see that the process IDs are increasing like crazy and though that this could be putting more strain to the server resources. – pataroulis Nov 22 '10 at 07:10
  • turn on slow query logging for mysql and check if you have bad queries or queries that need an index – Mike Nov 22 '10 at 13:12
0

Apparently my question was somewhat silly. Many thanks to Mike who helped me with the keepalive setting. This was the start to find what really was the problem.

The cURL script dedn't ever close the file descriptor so I had many MANY open files. Moreover, the network speed is crippled to 3MB/sec and this actually bottlenecks the system.

Since I started closing the open cURL file description, I have a feeling that only the network is the hog.

Another thing that I started changing is to drop from using MDB2. It sucks bigtime in terms of speed compared to native mysql_ calls.

Thanks a lot!

pataroulis
  • 143
  • 1
  • 2
  • 11
  • And here, sillyness prevailed. I just put eaccellerator for caching compiled files and the MDB overhead has disappeared.... – pataroulis Apr 14 '11 at 12:03