Troubled FastCGI Config -- 503 Errors

Question

I'm getting FastCGI hangups where some websites are accessible, but others return 503 Service Unavailable errors (if I wait long enough for a response). It's got lots and lots of domains on this host not accessible. This is a server with 1000 domains on it. It was running fine but slow occasionally until we switched to FastCGI. Then, it sped right up, but then many sites are not accessible. On two other servers we have, the FastCGI is running fine with 1000 domains and I cannot determine what's so different about this config.

This is CentOS 5. From the /etc/httpd/logs/error_log, I see these entries repeatedly. Perhaps they can provide clues as to what I need to do?

[warn] [client 50.50.50.50] mod_fcgid: can't apply process slot for /home/example/public_html/example.com/index.php
[warn] mod_fcgid: too much processes, please increase FCGID_MAX_APPLICATION
[error] [client 50.50.50.50] Premature end of script headers: index.php
[warn] [client 50.50.50.50] mod_fcgid: read data timeout in 400 seconds

Our /etc/httpd/conf/php.conf shows this arrangement:

# Fastcgi configuration for PHP5
LoadModule fcgid_module modules/mod_fcgid.so
MaxRequestsPerProcess 1000
#FcgidMaxProcesses 100
#FcgidProcessLifeTime 60
MaxProcessCount 500
IPCCommTimeout              400
IdleTimeout                 400
ProcessLifeTime 60
#DefaultMinClassProcessCount 120 
AddHandler fcgid-script .php5 .php4 .php .php3 .php2 .phtml
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php5
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php4
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php3
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php2
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .phtml

Answer 1

The process slot errors are resolved by identifying the server load bottleneck. In my case, it was disk I/O. I fixed that by running this command on the web directories. In this case, all the websites are on /home:

chattr -R +A /home

However, before you ever do that, be aware of the consequences:

http://tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap6sec73.html

and

http://en.wikipedia.org/wiki/Stat_%28Unix%29

Note that chattr only works on existing files. Newer files will not receive this attribute unless you rerun chattr. If you have hosted /home on another partition, then you could edit your /etc/fstab and add the noatime parameter on that partition.

You can also have a cron job that runs chattr on your /tmp folder every once in awhile in order to speed up things like PHP page execution (which usually caches in /tmp).

The other error regarding "[warn] mod_fcgid: too much processes, please increase FCGID_MAX_APPLICATION" -- that appears to be a bug that is filed with the Apache FastCGI team. Some people fix it by recompiling the source from the latest code. Some fix it by commenting out that error out of the C source and recompiling.