Apache bottle neck

Question

I have nginx as a front end proxy and apache with mod_fcgid.

This box has 24GB RAM and 16 cores. I shouldn't be bottlenecking.

There seems to be a bottleneck. Once it hits 7000 requests it hangs before continuing.

EDIT 3: Can someone explain PHP_FCGI_CHILDREN? I set it to 1 and stuff seems to work better. Does setting it higher better? It really spans a lot of processes like crazy higher it is.

EDIT: error log wise:

(104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server

EDIT - QUESTION 2 Because I am using nginx as a frontend proxy. Is it possible I am maxing out all my ports thus can no longer send to the backend(apache)? I also noticed that PHP procs die fast

16409 USER 20   0     0    0    0 Z 27.2  0.0   0:00.82 [php] <defunct>

Even with something like this it fails:

/usr/local/apache/bin/ab -n 1000000 -c 10 http://IP/index.php
apr_poll: The timeout specified has expired (70007)
Total of 23998 requests completed

My results:

/usr/local/apache/bin/ab -n 10000 -c 100 http:///index.php

Document Path:          /index.php
Document Length:        55764 bytes

Concurrency Level:      100
Time taken for tests:   52.076 seconds
Complete requests:      10000
Failed requests:        17
   (Connect: 0, Receive: 0, Length: 17, Exceptions: 0)
Write errors:           0
Non-2xx responses:      17
Total transferred:      558563436 bytes
HTML transferred:       556703249 bytes
Requests per second:    192.03 [#/sec] (mean)
Time per request:       520.764 [ms] (mean)
Time per request:       5.208 [ms] (mean, across all concurrent requests)
Transfer rate:          10474.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       6
Processing:     3  436 3978.0      3   52069
Waiting:        2  436 3978.0      3   52069
Total:          3  436 3978.1      3   52073

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      4
  90%      5
  95%      5
  98%   1018
  99%  29055
 100%  52073 (longest request)

/etc/sysctl.conf

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_reordering = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_max_syn_backlog = 2048
net.core.netdev_max_backlog = 1024
net.core.hot_list_length = 256
net.ipv4.tcp_max_tw_buckets = 360000
net.core.rmem_default = 65535
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.core.wmem_default = 65535
net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096 65535 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.optmem_max = 40960
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 2768    61000
fs.file-max = 209708

/usr/local/cpanel/cgi-sys/php5

export PHP_FCGI_CHILDREN=0
exec /usr/bin/php

/usr/local/apache/conf/php.conf

LoadModule fcgid_module modules/mod_fcgid.so
MaxRequestsPerProcess 1000
FcgidMaxProcesses 200
FcgidProcessLifeTime 3600
MaxProcessCount 200
FcgidIOTimeout 2000
FcgidIdleTimeout 60
FcgidIdleScanInterval 30
FcgidBusyTimeout 300
FcgidBusyScanInterval 80
ErrorScanInterval 3
ZombieScanInterval 3
FcgidMinProcessesPerClass 5
FcgidMaxProcessesPerClass 150
DefaultMinClassProcessCount 5
DefaultMaxClassProcessCount 150
IPCCommTimeout 40
IPCConnectTimeout 10
MaxRequestLen 1073741824
AddHandler fcgid-script .php5 .php4 .php .php3 .php2 .phtml
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php5
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php4
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php3
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php2
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .phtml

Apache

Timeout 5
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 15
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
ServerLimit 256
MaxClients 150
MaxRequestsPerChild 1000
KeepAlive Off
KeepAliveTimeout 1
MaxKeepAliveRequests 1

httpd -V

Server version: Apache/2.2.23 (Unix)
Server built:   Jan 29 2013 11:18:48
Cpanel::Easy::Apache v3.16.7 rev9999
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/experimental/event"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

nginx error log

2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15127 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15128 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15129 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15131 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15137 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15139 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15124 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15136 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15125 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15130 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received

Probably not what you are looking for but If you can replace `mod_fcgid` with `php-fpm` I would use the latter. Gives me more reliable results. It works with `mod_fastcgi` [tutorial](https://blogs.oracle.com/opal/entry/php_fpm_fastcgi_process_manager) or omit Apache and use `php-fpm` directly with nginx. — kei1aeh5quahQu4U, Feb 06 '13 at 20:24
Take a look at the [tutorial](https://blogs.oracle.com/opal/entry/php_fpm_fastcgi_process_manager). `php-fpm` has a lot of config files and options. There is some documentation here: http://php-fpm.org/wiki/Main_Page (probably outdated) and here http://php.net/manual/en/install.fpm.php. How hard? I think it depends on your application. If fcgid works for you I would not worry to switch. Plesk and Cpanel Integration does not exist, so you are on your own with php-fpm. The results you get could be caused by the `ab` benchmarking tool and not `mod_fcgid`. So don't switch if it's not critical. — kei1aeh5quahQu4U, Feb 06 '13 at 20:45
What do you mean by "Once it hits 7000 requests it hangs"? Your MaxClients is 150. Are you saying that if you send 7000 requests in /serial/ fashion to your webserver once it hits the 7000th request it just hangs? e.g. ab -c 1 -n 7000 http://www.example.com — HTTP500, Feb 06 '13 at 21:25
It went from 1000-7000 real quick and then it just sat on 7000/8000 before finishing. — Tiffany Walker, Feb 06 '13 at 21:26
Try setting your StartServers to 150 (equal to max clients). — toppledwagon, Feb 06 '13 at 22:06
Are there corresponding entries in /var/log/messages? What version of nginx are you using? nginx -v — HTTP500, Feb 07 '13 at 00:16
Tried as mod_php. apr_socket_recv: Connection timed out (110) — Tiffany Walker, Feb 07 '13 at 17:11

score 0 · Answer 1 · answered Feb 06 '13 at 21:30

0

Are you running the benchmark on a separate machine to the server? What's the network link speed on the server? You could be maxing out the network bandwidth.

Transfer rate:          10474.46 [Kbytes/sec] received

This looks suspiciously like 10MB/s

answered Feb 06 '13 at 21:30

chrskly

1,569
12
16

This was being ran on the same machine – Tiffany Walker Feb 06 '13 at 21:33

score 0 · Answer 2 · answered Feb 06 '13 at 21:44

0

The percentage of requests served within 5 seconds is 95, then all of a sudden jumps to 1018 seconds to server just 98%. It also happens that your Apache Timeout is set to 5 seconds. I think the default Timeout value for Apache is generally set to be greater than 120 seconds by default. Is there a reason why you set it to 5 seconds? Have you considered increasing it?

 95%      5
 98%   1018

answered Feb 06 '13 at 21:44

Daniel t.

9,291
1
33
36

Tried and same results. – Tiffany Walker Feb 06 '13 at 21:51
Your MaxClients seems very low for a server of such capacity. May be you need to increase that, then restart the service - just to remind you - many of us forget that. – Daniel t. Feb 06 '13 at 22:32
Set 1024. It looks to be more PHP related then Apache related. – Tiffany Walker Feb 06 '13 at 22:39

score 0 · Accepted Answer · answered Feb 07 '13 at 00:08

0

If you already use ngix for a reverse proxy and I assume also for serving static files, then mod_fcgi is just a complication — just use apache with mod_php and reasonably high maxclients and serverlimit (something like 1024).

answered Feb 07 '13 at 00:08

Tometzky

2,679
4
26
32

So I really don't need mod_fcgid for performance in anyway? So running as DSO would be faster and quicker and easier? – Tiffany Walker Feb 07 '13 at 00:09
I'm pretty sure you don't. The problem with running apache+php directly is that keepalive connections and static requests use resources that are very limited, like httpd processes in prefork-mpm. And you have to use prefork if you need PHP. But when you hava a ngix as a reverse proxy and it is serving static files, then you have no problems. I even run apache in worker-mpm mode as a reverse proxy for several other apache instances, each running another PHP site with mod_php and prefork and each running as it's own UID for security isolation. – Tometzky Feb 07 '13 at 07:46

Apache bottle neck

3 Answers3