2

Apache server Debian Linux x64 serves the site mono applications in mysite.com and other sites using virtual hosting.

Sometimes apache stops responding for unknow reason. Apache error_log contains:

[Sat Jun 20 13:56:31 2015] [error] [client 90.190.33.164] File does not exist: /var/www/apple-touch-
icon.png
[Sat Jun 20 16:38:04 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sat Jun 20 17:20:11 2015] [notice] caught SIGTERM, shutting down
mod-mono-server received a shutdown message
[Tue Jun 23 09:20:50 2015] [error] mod_mono: connect error (Interrupted system call). File: /tmp/mod
_mono_server_default
mod-mono-server received a shutdown message
mod-mono-server received a shutdown message
mod-mono-server received a shutdown message
[Tue Jun 23 09:20:54 2015] [error] Not running mod-mono-server.exe because no MonoApplications, Mono
ApplicationsConfigFile or MonoApplicationConfigDir specified.
[Tue Jun 23 09:20:54 2015] [notice] Apache/2.2.16 (Debian) mod_mono/2.6.3 configured -- resuming normal operations
..

It looks like

[Sat Jun 20 16:38:04 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting

message in log file appears first.

After that probably cron job tries below to re-start apache which causes message in error_log:

[Sat Jun 20 17:20:11 2015] [notice] caught SIGTERM, shutting down
mod-mono-server received a shutdown message

Server was down starting at this time: access_log does not log any access starting from Jun 20 17:20

Server starts resoping only at [Tue Jun 23 09:20:50 2015] when I manually re-started it. This start to occur from last month several times per month.

To fix this the following crontab script is used running after every 10 minutes:

wget --timeout=10 --no-verbose --tries=1 -a /var/log/wget.log -O /var/log/wgettulem.html  mysite.com || /etc/init.d/apache2 restart

If apache stops responding, log file written by this script contains

Connection timed out error messages.

However apache is not restarted. Trying to re-start it manually using

/etc/init.d/apache2 restart

returns Sockect is in use error. ps aux shows that apace is stil running.

I fix this by using

ps aux | grpe apache  

to find appache process number and use

kill

to kill this process.

How to fix or find the reson of this message ? How to automate this process so that all apache instances are killed and apache will be re-started automatically by this script. How is there bettor way to keep apache responding ?

Debian version is 6.0.4

apache2 -V returns

Server version: Apache/2.2.16 (Debian)
Server built:   Feb  5 2012 21:35:42
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

apache2.conf contains

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
Andrus
  • 169
  • 4
  • 12
  • have you tried actually fixing the error instead of the restart because of the issue? server reached MaxClients setting, consider raising the MaxClients setting – Dennis Nolte Jun 23 '15 at 08:46
  • Maybe mono application hangs or terminates in unusual way and mod_mono leaves apache connection open. Application is written by me but I dont have an idea how to fix this. It started to occur about month ago. Switching to java or php is huge work. CPU usage was 0%. Since apache is not responding, no way to get information. Maybe some shell script which invokes kill for apache process can used. – Andrus Jun 23 '15 at 11:30

1 Answers1

0

If you merely want to forcibly restart Apache if it hangs (for any reason) then you should install System Integrity Monitor by R-fx Networks.

Among many features, it can be configured to monitor your web services, automatically issue restart commands and send out notifications.

Quick install instructions:

cd /usr/local/src
wget http://www.rfxn.com/downloads/sim-current.tar.gz
tar -xvf sim-current.tar.gz
cd sim*
./setup -i

Proceed with the interactive installation defining the appropriate filepaths for your web service executables. As a last step, enable the cronjob:

/usr/local/sbin/sim -j
Elliot B.
  • 1,356
  • 2
  • 18
  • 28
  • Is'nt it simpler to use some shell script which invokes kill for apache process ? It looks like it issues simple `/etc/init.d/apache2 restart` which didnt work in my case. It is same as script in my answer – Andrus Jun 23 '15 at 11:33
  • @Andrus You can configure SIM to invoke a custom script when an event occurs -- such as a scripts that forcibly kills off Apache. There are many things that SIM has taken into account that would be tiresome to re-code in your own shell scripts. For instance, SIM can be configured to stop restarting services when it was made X number of restart attempts in X amount of time. You wouldn't want your shell script to endlessly restart Apache. Furthermore, SIM is all setup to page out notifications in the event of failure. – Elliot B. Jun 23 '15 at 16:38
  • I found that `killall -w apache2` probably does what I need. So `cron +wget + killall + apache2 restart` seems sufficient for me – Andrus Jun 23 '15 at 18:07
  • Why are you recommending 3rd party downloads? Maybe use what's already available (repo) and works well: MONIT comes to mind. It will monitor/restart etc. http://manpages.ubuntu.com/manpages/bionic/man1/monit.1.html `sudo apt install monit` IIRC apache2 is already configured to restart on sigterms under default install of Ubuntu build for Monit. or see: https://mmonit.com/wiki/Monit/ConfigurationExamples#apache – B. Shea Aug 17 '18 at 12:58
  • Why not recommend a third party solution? I recommended SIM 3.5 years ago because it works well and addresses the OP's needs. Monit is another great solution. However, you do realize that monit is also a "third party" download? Just because the package is part of the base yum repos, doesn't mean it was written by or supported by the same group that built the operating system. – Elliot B. Aug 17 '18 at 15:36