3

Every sunday night during debian squeeze's logrotation, apache crashes when postlogrotation attemts to reload configuration, and/or restart apache. I've even tried reconfiguring the logrotation by hard restarting (stop start) the service, with the same faults.

What happends is that apache acctually seems to be starting fine, but every request throws the following error in apaches error.log:

[notice] child pid xxxx exit signal Segmentation fault (11)

If I restart apache manually once more, it works fine again. Does anyone have any idea why this is happening?

The AWS instance is running debian squeeze, apache 2.2.16, php 5.4.32 and varnish 3 (if it matters).

Any suggestions is very welcome. If theres relevant info I've missed, tell me!

UPDATE: I moved the Magento sites to another server, and the problem stopped occurring (even though there are still a few other sites left), and it didn't happen on the new server. Still few ideas as to what caused it.

Harald
  • 33
  • 5
  • Smells like a funky environment problem. What does `env` print when you run it in your interactive shell? – womble Oct 27 '15 at 09:09
  • Do you see anything special/wrong, @womble? Sorry for bad formatting.. `root@:~# env TERM=xterm-256color SHELL=/bin/bash SSH_CLIENT= 57029 22 SSH_TTY=/dev/pts/0 USER=root MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/root LANG=nb_NO.UTF-8 SHLVL=1 HOME=/root LOGNAME=root SSH_CONNECTION= 57029 xx.xx.xx.xxx 22 LC_CTYPE=UTF-8 _=/usr/bin/env` – Harald Oct 30 '15 at 13:17
  • @Harald, did you actually try to run the logrotate script manually in verbose mode to see if there is something? If not, try this: `logrotate -vf /etc/logrotate.d/apache2` and check the output. – Diamond Dec 09 '15 at 10:07
  • Is there anything unusual about the directory where the logs are stored? E.g. some sort of remote mount? – mc0e Dec 14 '15 at 01:33
  • Having the same problem with two servers after having them updated to Debian Wheezy (no problems with Squeeze before). Several Magento websites, but it is only one specific site on each server that causes the segfault, until I reload or restart Apache once or twice. CoreDumps are not created - no idea why as the same procedure works on a testing VM. I suspect APC, but couldn't examine this yet. – Larsen Mar 14 '16 at 11:08

2 Answers2

4

There can be different reasons for such a failure and rather then assuming something, I would suggest that you take a little time to debug both the logrotate script and make a coredump of apache to analyse the issue with gdb.

You can manually run the logrotate script in verbose mode like this (-v verbose, -f force):

logrotate -vf /etc/logrotate.d/apache2

For making a cordump and analyse it with gdb, you can install all the necessary packages using apt-get and it's not very difficult to configure.

Install gdb and necessyry symbol files::

apt-get install gdb 
apt-get install apache2-dbg php5-dbg

Set CoreDumpDirectory in apache config:

CoreDumpDirectory /tmp/apache2-coredump

Set ulimit to unlimited:

ulimit -c unlimited

You can read the coredump with gdb like this:

gdb apache2 -core /tmp/apache2-coredump/core

Here is a detailed how-to: How to figure out what is causing an Apache Segmentation Fault

Diamond
  • 9,001
  • 3
  • 24
  • 38
  • 2
    Given that the segmentation fault doesn't happen till you get a request, you have a chance to try attaching tools to your apache child processes to try to catch the event happening. e.g. `strace`, `ltrace`, `gdb`. All of those can be attached to a running process, so it might be easier to attach to the child process that way rather than running the parent process under gdb. – mc0e Dec 11 '15 at 15:08
  • The first thing I'd try would be to find the busiest apache child, connect to it with `strace -p ` and hit a few pages till you see that process get the request. Tell us what the process was trying to do when it died. If it's not clear, do the same with ltrace. Failing that, gdb is more powerful, but also more work and more learning required. – mc0e Dec 14 '15 at 01:38
2

Change rotation method by using rotatelogs in you apache configuration :

 CustomLog "|bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" common

rotatelogs will do the log rotation, so no graceful is needed anymore.

You cron will only need to compress old logs.

The documentation : https://httpd.apache.org/docs/2.2/programs/rotatelogs.html

PS : I know that this does not explain why you have segmentation fault, but with this method you will not have graceful anymore during rotation.

mick
  • 735
  • 6
  • 7