1

For some reason, most Sundays around 5-6AM local time Apache is not working properly. I can't find the reason, and I'm stuck troubleshooting.

I'm running Magento on Apache2 with an SSL certificate, and around that specific time random pages stop functioning resulting in no orders being placed. Sometimes it's the category page, sometimes it's the admin area.

I'm thinking it may be logrotating, but not sure how to troubleshoot further. Here's my Apache error log for the past two Sundays:

Jan 29th

/var/log/apache2/error.log.1:[Sun Jan 29 00:35:21 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting

/var/log/apache2/error.log.1:[Sun Jan 29 00:39:44 2012] [error] [client 66.87.100.194] request failed: error reading the headers

/var/log/apache2/error.log.1:[Sun Jan 29 06:25:21 2012] [notice] Graceful restart requested, doing restart

--- notice new log file

/var/log/apache2/error.log:[Sun Jan 29 06:25:24 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations

/var/log/apache2/error.log:[Sun Jan 29 06:26:00 2012] [notice] child pid 7301 exit signal Segmentation fault (11)

Jan 22nd

/var/log/apache2/error.log.2.gz:[Sun Jan 22 06:25:17 2012] [notice] Graceful restart requested, doing restart

--- notice new log file

/var/log/apache2/error.log.1:[Sun Jan 22 06:25:22 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations

/var/log/apache2/error.log.1:[Sun Jan 22 16:26:11 2012] [notice] caught SIGTERM, shutting down

/var/log/apache2/error.log.1:[Sun Jan 22 16:26:12 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations

/var/log/apache2/error.log.1:[Sun Jan 22 16:26:14 2012] [notice] child pid 30367 exit signal Segmentation fault (11)

Community
  • 1
Rune
  • 11
  • 2

3 Answers3

1

Not quite a programming problem, but there's this

server reached MaxClients setting, consider raising the MaxClients setting

which indicates Apachex is trying to take on more requests (at once) than it's configured for. Upping this number may be a good idea.

There's also these two lines 

/var/log/apache2/error.log:[Sun Jan 29 06:25:24 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations

/var/log/apache2/error.log.2.gz:[Sun Jan 22 06:25:17 2012] [notice] Graceful restart requested, doing restart

Which indicate someone is upgrading server packages, which may interfere with performance of the server itself.

ServerFault's a good place to ask a question like this.

A better place would be your ISP/IT-Department/Web-Host.

Alana Storm
  • 458
  • 5
  • 16
  • I don't believe maxclients is the issue as it doesn't appear in the log on every crash. Also, it seems to be a notice, and not an error causing unresponsiveness at the point in time. – Rune Feb 01 '12 at 09:39
1

Do you have any cronjobs running at that same time? My hunch is that there is a system update running that updates some shared library loaded by your apache app, which will often cause segfaults.

evil otto
  • 151
  • 2
  • +1 My hunch is that the Sunday "graveyard" is a slot where the sysadmins are doing backups and bouncing Apache. Talk to the IT support group, and understand the constraints here that you need to work with. – TerryE Jan 31 '12 at 13:19
  • This seems likely actually. Going to contact my hosting provider and ask. Will let you know. – Rune Feb 01 '12 at 09:38
  • Apparently backups etc is running 5 hours before. It's the exact same time as logrotate etc., but it's running daily so I don't see why it should only have an impact on Sunday? – Rune Feb 01 '12 at 12:23
0

I did research on google about "segmentation fault" error, most of them related with 3rd party applications which is can be eAccelerator, iconcube, zend encoder etc. Also, somebodies written a comment that can be related with PHP edition. Their suggestion, uninstall the whole PHP then re-install again.

My suggestion, if you have eAccelerator, remove for a while and look up the log again. If error keep going, try to install fresh PHP without "Suhosin" patch.

Segmentation Fault

Community
  • 1
Oğuz Çelikdemir
  • 151
  • 1
  • 3
  • Would segmentation fault result in unresponsive pages as I experience? I'm not using eAccelerator and reluctant to re-install PHP on a live server. – Rune Feb 01 '12 at 12:25
  • I've found that APC can cause segmentation faults. http://2bits.com/articles/php-op-code-caches-accelerators-a-must-for-a-large-site.html under "Drawbacks of PHP op-code caches: Segmentation Faults". Could that explain my issues? – Rune Feb 01 '12 at 13:12
  • @Rune as I said in my post, you should try step by step if you are not sure which application has side effects. The interesting part in your issue "the problem raise in specified times", that is interesting. For instance, you can check Linux log file from related days like `vi /var/log/messages`, `vi /var/log/cron.log` – Oğuz Çelikdemir Feb 01 '12 at 15:55
  • I'll try to narrow it down. For now I've tweaked APC. I've looked through the logs and all I find around that specific time is this (linking to pastebin) http://pastebin.com/ZctmSLjr – Rune Feb 02 '12 at 11:01
  • check this link : [CRON: Regular Tasks](http://www.linuxtopia.org/online_books/linux_beginner_books/debian_linux_desktop_survival_guide/CRON_Regular.shtml) – Oğuz Çelikdemir Feb 02 '12 at 11:49
  • and that is interesting [Syslog Daemon Problem](http://www.linuxquestions.org/questions/linux-software-2/syslog-daemon-problem-rsyslogd-was-huped-904150/) – Oğuz Çelikdemir Feb 02 '12 at 11:54
  • [Rsyslog Huped](http://ubuntuforums.org/showthread.php?t=1384521) – Oğuz Çelikdemir Feb 02 '12 at 11:55
  • After tweaking APC I didn't see any down time this Sunday. I'll keep monitoring and update this thread if I experience more issues. Thank you for your help so far! – Rune Feb 07 '12 at 13:41