Degraded httpd performance after moving from RHEL 5.6 to 6.1

Question

We have recently replaced an RHEL 5.6 web server with RHEL 6.1. For both environments, the (stock) Redhat shipped Apache httpd has been used (i.e. yum install httpd). The server is serving PHP content and is busy (serving approximately 2500 - 4000 page requests per minute). The specifications of both servers are identical in terms of memory, storage and network connection. What we are seeing is significantly higher load averages on the RHEL 6.1 box - where the load average will (at times) spike to over 40 (all httpd processes) resulting in substantially degraded performance of the site. We have monitored the RHEL 5.6 environment and the load average does not exceed approximately 5 concurrent httpds. How can we investigate this issue ? Please bear in mind that this is a production environment, but we can compare "apples with apples" by switching between the 5.6 and 6.1 server.

score 4 · Answer 1 · answered Oct 27 '11 at 22:31

Upgrading from 5.6 to 6.1 is a big step - you're probably running different versions of both apache, php, and perhaps a bytecode cache as well.

If you're using a bytecode cache (which I believe you should, with the kind of load you're describing), you should verify that it's working (I know apc has changed some configuration syntax between versions, for example). Which one are you in that case using?

You should have some metrics explaining what kind of cpu time is spent - if it's IOWait, System or User time. The same applies to memory usage, etc. How does these things look?

I will again recommend some kind of monitoring tool, like for example Munin.

Thanks very much for this. Yes we are using eAccelerator as our bytecode cache. I'm not certain on the configuration of this bytecode cache (in particular whether it's identical between the RHEL 5.6 and 6.1 system). This is something for us to follow up - helpful reply. — Patrick Rynhart, Oct 27 '11 at 23:21

score 2 · Accepted Answer · answered Oct 28 '11 at 09:05

Your 2500 - 4000 requests per minute translates to 40-60 requests per second. That kind of load hardly needs kernel level tuning, but more likely there is something wrong with your Apache or PHP setup. Some typical reasons include

Long TimeOut values in httpd.conf
KeepAlive on and/or long KeepAliveTimeOut values in httpd.conf (can lead to extra httpd processes)
Some unnecessary httpd module loaded
Some unnecessary PHP module loaded
Misconfigured memcached setup (if used)
Database connectivity/configuration problem

You need to figure out what's different compared to your old server. Did you copy the configuration values from the old server to new one? Was the old server fine-tuned years ago and you have forgotten something vital?

What does Apache's server-status page show you?

If everything else fails, you can always use PHP's XDebug module. That means you perform page loads at your server, and let XDebug generate a Valgrind compatible report for you. Then you can analyze that file with KCacheGrind or some other analyzer and see where the precious CPU time gets consumed. That can give you a clue about what's wrong.

Thanks Janne. Re: "You need to figure out what's different compared to your old server. Did you copy the configuration values from the old server to new one? Was the old server fine-tuned years ago and you have forgotten something vital?" You've hit the nail on the head - the server was set-up by a former staff member. We believe that we have copied the configuration across correctly, but it looks like the old server was fine tuned. Thanks for the XDebug+valgrind approach - this looks useful and I will try to follow this up. — Patrick Rynhart, Oct 31 '11 at 07:48

score 1 · Answer 3 · answered Oct 27 '11 at 22:23

1

Perhaps you can compare the output of sysctl -A on both systems? You may have tuned your RHEL5 kernel. I've also found oprofile to be useful to determine where the kernel and processes are spending time.

Edit: I assume apache and php are configured the same way.

answered Oct 27 '11 at 22:23

Mark Wagner

18,019
2
32
47

Thanks for this helpful reply which we will follow up. Apache & PHP are configured identically - we have been focusing on the httpd config (i.e. threads vs processes etc) and hadn't thought of the kernel config. – Patrick Rynhart Oct 27 '11 at 23:19

score 1 · Answer 4 · answered Oct 27 '11 at 22:24

1

Question: Is the both apache configurations the same i.e. are you using thread invokations or process invokations for each request?

If you are using process invokations, then a process is created for each request (with a few remaining behind) and thus a higher load factor.

If you are using thread invokations, then a number of requests will be handled by threads within a far fewer number of processes.

Thus, there might be nothing wrong.

answered Oct 27 '11 at 22:24

mdpc

11,856
28
53
67

Both the old and the new server were configured using the default processing model (MPM) which is the process-based prefork model. However, we have since been reading up on "prefork vs worker" and are a bit confused as to which we should be using. Worker seems to have lower memory requirements, but this is not a problem with our environment (i.e. RAM is not a limitation). – Patrick Rynhart Oct 27 '11 at 23:37
I don't know if this still applies, but a while ago it was advised not to use worker mpm and php as a module because the latter was (and may still) is not thread-safe. – AndreasM Oct 28 '11 at 09:05

score 1 · Answer 5 · answered Oct 28 '11 at 08:49

While a lot depends on what the PHP is actually doing, IMHO the load average seems to be rather high even on the old machine - but obviously the pressing issue is the difference in behaviour between old and new.

Are you sure that the configs are same? Have you checked the apache error logs to see if its having any problems with any of the processing directives? Have you checked the permissions on the eaccelerator dir?

Does the box use external resources (such as DNS, database)? If so, is it on the same network as the old one? Are the RTT times to the other servers similar?

Can you easily downgrade the box to RH5.6? If so, does this resolve the problem?

Are you seeing a change in the bandwidth or request response times (%D)?

Worker seems to have lower memory requirements

The difference isn't really all that great - since the bulk of the memory is in TXT segments marked as COW. Indeed, some benchmarks show pre-fork to be more scalable than worker threads on Linux.

Thanks for this (this whole thread has provided excellent suggestions). We believe that the configs are the same and that eAccelerator is working. The box does use DNS+DB, but is also on the same network. Thank you for commenting on worker vs pre-fork also. — Patrick Rynhart, Oct 31 '11 at 07:53

Degraded httpd performance after moving from RHEL 5.6 to 6.1

5 Answers5