PHP-FPM Can't create enough children and stops serving requests

Question

I'm new to PHP-FPM and I've noticed at random intervals it will seemingly just lock up.

[26-Dec-2014 22:50:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 921 total children
[26-Dec-2014 22:50:17] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 931 total children
[26-Dec-2014 22:50:18] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 941 total children
[26-Dec-2014 22:50:19] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 951 total children
[26-Dec-2014 22:50:20] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 961 total children
[26-Dec-2014 22:50:21] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 971 total children
[26-Dec-2014 22:50:22] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 981 total children
[26-Dec-2014 22:50:23] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 991 total children
[26-Dec-2014 22:50:24] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 1001 total children
[26-Dec-2014 22:50:25] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 1011 total children
[26-Dec-2014 22:50:26] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 1021 total children
[26-Dec-2014 22:50:27] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 9 idle, and 1031 total children

This happens seemingly at random (we could have 50 people on the site per minute or 200) when it does happen PHP-FPM stops serving requests (by looking above it looks like it can't meet demand)

Even if I restart PHP-FPM it remains locked up for a short period.

[26-Dec-2014 22:52:41] NOTICE: Terminating ...
[26-Dec-2014 22:52:41] NOTICE: exiting, bye-bye!
[26-Dec-2014 22:52:41] NOTICE: fpm is running, pid 17484
[26-Dec-2014 22:52:41] NOTICE: ready to handle connections
[26-Dec-2014 22:53:15] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 53 total children
[26-Dec-2014 22:53:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 61 total children
[26-Dec-2014 22:53:17] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 71 total children
[26-Dec-2014 22:53:18] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 81 total children
[26-Dec-2014 22:57:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 109 total children
[26-Dec-2014 22:58:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 124 total children
[26-Dec-2014 22:58:10] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 132 total children
[26-Dec-2014 22:58:11] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 142 total children
[26-Dec-2014 22:58:12] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 152 total children
[26-Dec-2014 22:58:13] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 162 total children
[26-Dec-2014 22:58:14] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 172 total children
[26-Dec-2014 22:58:15] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 182 total children
[26-Dec-2014 22:58:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 192 total children
[26-Dec-2014 22:58:17] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 202 total children

Here are my settings

pm.max_children = 2000
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 200
pm.max_requests = 500

I have tried these also at

pm.max_children = 2000
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 50
pm.max_requests = 500

and a few other combinations but still see the issue.

Server is 32 core, 64GB memory. Using Apache and FastCGI

Thanks

a flaw in php5-fcgi is being used to prevent your server from ever terminating the request. this vulnerability was reported earlier today. basically, you use stream select to connect to the remote server but never close the socket resulting in the process hanging. the attacker simply repeats this using a for loop until your server exhausts its resources. you might consider moving over to an nginx webserver. see more here: https://github.com/php/php-src/pull/966 — r3wt, Dec 26 '14 at 23:25
@r3wt reading through, that appears only if the attacker connects directly to the PHP-FPM host directly which isn't the case here. The server is also behind a transparent proxy. — TheBritishAreComing, Dec 26 '14 at 23:47

score 1 · Answer 1 · answered Dec 27 '14 at 12:06

try this:

Note: Used only when pm is set to 'dynamic'

pm.start_servers = min_spare_servers + (max_spare_servers - min_spare_servers) / 2

pm = dynamic
pm.max_children = 500
pm.start_servers = 60
pm.min_spare_servers = 20
pm.max_spare_servers = 100
pm.max_requests = 10000;

or like this:

pm = ondemand
pm.max_children = 500
pm.max_requests = 10000

anyway you need to use cache.

PHP-FPM Can't create enough children and stops serving requests

1 Answers1