3

My Apache status looks like; 

201 requests/sec - 98.8 kB/second - 504 B/request
85 requests currently being processed, 345 idle workers
_____CCW_C_____C__C__C_R____C_WC_________C__C____CW__C__CCC_____
__C____W______C___C___CW__C_C______C__W_C__C_____CCC____C______R
CC_C_______C___C____C______________C______C__C________________C_
___________________C______________________C_______C___C_____C___
CC____C__C___R_____C_C_CC__________C___C___________R____C_C_C___
______C______W_W__W___C____________________C__WCC__R__R_C_______
R__RC________________________C___R____W__C____..................
....................................................

Server load is average 2 on a 4 core machine.

IO utilization is 10-15% and doesn't have many jumps over 70%.

Machine has almost 4 gb free and uses 0 swap.

The site on the machine is a PHP site. All PHP code is optimized and fast mostly when it gets accessed, however sometimes requests get stuck. Stuck meaning; no response for at least 10 sec. We debugged the PHP code, but it is quite optimal and fast. We spend a lot of time on it until we decided to test the requesting of: 

<html><body>test</body></html>

test.html page.

This static resource also gets 'stuck' in the same manner the php pages get 'stuck'.

How is that possible given the health of the system and the fact that it is a static file?

I tested the network, but, when the PHP shows 'slowness' in the site monitoring, the html test files also take (far longer) than 10 sec to load using;

time lynx -dump http://127.0.0.1/test.html

We are kind of desperate to solve this problem, but we cannot seem to tackle it.

user45339
  • 53
  • 7

2 Answers2

1

Maybe Apache runs out of file handles? How many file handles you have allowed it to have? The default 1024 can be a bottleneck quite soon. In Linux you up the limits in /etc/security/limits.conf file.

Is there lots of disk activity during the stalls? If you have Apache access log and other logs very verbosely enabled, maybe it's the filesystem committing the latest changes? That should not affect Apache in any way, but you never know.

And just to make sure, take a look at /proc/sys/kernel/random/entropy_avail during the stalls. You can see it with for example watch -n1 'cat /proc/sys/kernel/random/entropy_avail'. If it says 0, your kernel has ran out of entropy and that blocks Apache until there is more entropy available.

If that's the case you can install rng-tools and run the rngd daemon, which shovels semi-random numbers from /dev/urandom to /dev/random in the situations where the real entropy is not available.

Janne Pikkarainen
  • 31,852
  • 4
  • 58
  • 81
0

I haven't dug into the internals, but from my experience and from what I'm told... if PHP is running on Apache using the embedded module (libphp5.so) then Apache loads PHP (and optionally, any shared modules) into memory on every request, even if PHP code is not being run.

Consider using nginx as a reverse proxy in front of Apache. nginx is incredibly fast at serving static resources, and if configured correctly can really reduce load on a busy web server. For bonus points, configure PHP to run via FastCGI within nginx. Take a look at this article to discover some reasons why. It's a really great way to go. I set up a new web server with Ubuntu 10.04, nginx, spawn-fcgi and php-cgi just last week and it took almost no time at all. PHP 5.3 is bundled with Ubuntu 10.04, for the record.

patcoll
  • 146
  • 2
  • Hi, You are right about Nginx; we now put it in front for the static stuff. Still Apache has problems. We also tried FastCGI in nginx but that is *so* much slower than libphp5; it completely burns the server into the ground. It'll just refuse to serve anything basically if we do that. – user45339 Jun 13 '10 at 12:18
  • That's great you've been able to rule that out. I'd say a next step would be to look at the shared modules (if any) that PHP loads. Remove all the modules you don't need, especially in production. Unfortunately you're out of luck if those modules are compiled into the library itself (not loaded in as shared) -- you'll have to recompile PHP in that case. From an optimization perspective, if all code that will run on that server will only use a select few PHP modules, it may be a good idea to recompile PHP and build those modules into `libphp5.so` itself. – patcoll Jun 17 '10 at 15:59