1

We are running Apache 2.2.15 and PHP 5.3.3

We have intermittent occurrences of Apache getting overloaded with processes stuck in W - sending reply status. This seems to happen at random and will last for around an hour or so. We do not do anything to the server and the problem will go away. As I mentioned, when the problem happens, running “apachectl status” will show most clients stuck in “W” – sending reply status. MySQL will also show a bunch of connections in sleep state. New relic will then show that redis response time goes extremely high. We are using redis for magento cache.

We've checked the apache error logs and nothing is in there that is out of the ordinary. It has been recommended to us to upgrade Apache to 2.2.31 and PHP to 5.5 but we aren't sure that it will fix the problem.

Any ideas? We did an strace on a PID when the issue was happening and here what it showed. This just repeated and repeated:

poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$32\r\nzc:k:e22_Zend"..., 60, MSG_DONTWAIT, NULL, 0) = 60
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$175\r\na:7:{s:3:\"sun\";s:6:\"Sunday"..., 8192, MSG_DONTWAIT, NULL, NULL) = 183
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$34\r\nzc:k:e22_Zend"..., 62, MSG_DONTWAIT, NULL, 0) = 62
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$216\r\na:12:{i:1;s:7:\"January\";i:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 224
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$62\r\nzc:k:e22_Zend"..., 90, MSG_DONTWAIT, NULL, 0) = 90
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$178\r\na:12:{i:1;s:3:\"Jan\";i:2;s:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 186
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\343\0\0\0\3SELECT COUNT(DISTINCT e.ent"..., 231) = 231
read(26, "\1\0\0\1\0011\0\0\2\3def\0\0\0\33COUNT(DISTINCT "..., 16384) = 83
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "[\0\0\0\3SELECT `catalog_category_en"..., 95) = 95
read(26, "\1\0\0\1\n]\0\0\2\3def\7magento\27catalog_ca"..., 16384) = 1085
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\n\7\0\0\3SELECT `attr_table`.* FROM "..., 1806) = 1806
read(26, "\1\0\0\1\6&\0\0\2\3def\0\0\0\10value_id\10value_"..., 16384) = 961
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$32\r\nzc:k:e22_Zend"..., 60, MSG_DONTWAIT, NULL, 0) = 60
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$175\r\na:7:{s:3:\"sun\";s:6:\"Sunday"..., 8192, MSG_DONTWAIT, NULL, NULL) = 183
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$34\r\nzc:k:e22_Zend"..., 62, MSG_DONTWAIT, NULL, 0) = 62
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$216\r\na:12:{i:1;s:7:\"January\";i:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 224
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$62\r\nzc:k:e22_Zend"..., 90, MSG_DONTWAIT, NULL, 0) = 90
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$178\r\na:12:{i:1;s:3:\"Jan\";i:2;s:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 186
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\343\0\0\0\3SELECT COUNT(DISTINCT e.ent"..., 231) = 231
read(26, "\1\0\0\1\0011\0\0\2\3def\0\0\0\33COUNT(DISTINCT "..., 16384) = 83
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\\\0\0\0\3SELECT `catalog_category_en"..., 96) = 96
read(26, "\1\0\0\1\n]\0\0\2\3def\7magento\27catalog_ca"..., 16384) = 1082
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\20\7\0\0\3SELECT `attr_table`.* FROM "..., 1812) = 1812
read(26, "\1\0\0\1\6&\0\0\2\3def\0\0\0\10value_id\10value_"..., 16384) = 966
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$32\r\nzc:k:e22_Zend"..., 60, MSG_DONTWAIT, NULL, 0) = 60
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$175\r\na:7:{s:3:\"sun\";s:6:\"Sunday"..., 8192, MSG_DONTWAIT, NULL, NULL) = 183
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$34\r\nzc:k:e22_Zend"..., 62, MSG_DONTWAIT, NULL, 0) = 62
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$216\r\na:12:{i:1;s:7:\"January\";i:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 224
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
sendto(25, "*3\r\n$4\r\nHGET\r\n$62\r\nzc:k:e22_Zend"..., 90, MSG_DONTWAIT, NULL, 0) = 90
poll([{fd=25, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0 (Timeout)
poll([{fd=25, events=POLLIN|POLLERR|POLLHUP}], 1, 10000) = 1 ([{fd=25, revents=POLLIN}])
recvfrom(25, "$178\r\na:12:{i:1;s:3:\"Jan\";i:2;s:"..., 8192, MSG_DONTWAIT, NULL, NULL) = 186
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("/usr/share/zoneinfo/America/Chicago", {st_mode=S_IFREG|0644, st_size=3543, ...}) = 0
stat("/usr/share/zoneinfo/UTC", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\376\0\0\0\3SELECT COUNT(DISTINCT e.ent"..., 258) = 258
read(26, "\1\0\0\1\0011\0\0\2\3def\0\0\0\33COUNT(DISTINCT "..., 16384) = 83
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "[\0\0\0\3SELECT `catalog_category_en"..., 95) = 95
read(26, "\1\0\0\1\n]\0\0\2\3def\7magento\27catalog_ca"..., 16384) = 1087
poll([{fd=26, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(26, "\n\7\0\0\3SELECT `attr_table`.* FROM "..., 1806) = 1806
read(26, "\1\0\0\1\6&\0\0\2\3def\0\0\0\10value_id\10value_"..., 16384) = 956
Froggiz
  • 3,043
  • 1
  • 19
  • 30
Jeremy Miller
  • 11
  • 2

0 Answers0