Apache child process does not exit and is not terminated

Question

I am running a web application with Debian 9, Apache 2.4, mod_wsgi 4.5, django 1.11 and python 2.7.

My apache conf is

ServerTokens Prod
ServerSignature Off
FileETag MTime Size
Header set X-Frame-Options SAMEORIGIN
<Location />
    <LimitExcept HEAD GET POST PUT DELETE>
        order deny,allow
        deny from all
    </LimitExcept>
</Location>

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

WSGIRestrictEmbedded On

<VirtualHost *:443>
    RewriteEngine On
    RewriteCond %{HTTP_HOST} ^www\.<server name>$
    RewriteRule ^(.*)$ https://<server name>/ [R=301,L]

    SSLEngine On
    SSLProtocol +TLSv1.1 +TLSv1.2
    SSLHonorCipherOrder On
    SSLCertificateFile /path/to/cert.crt
    SSLCertificateKeyFile /path/to/key.key
    #Change needed in release..This resolves cert issue.
    SSLCertificateChainFile /path/to/ca.ca-bundle

    ServerName <server name>
    ServerAlias <secondary server name>

    DocumentRoot /path/to/root

    #Alias /robots.txt /path/to/root/robots.txt
    #Alias /favicon.ico /path/to/root/images/favicon.ico
    #Alias /fonts/ /path/to/root/fonts/
    Alias /images/ /path/to/root/images/
    Alias /style/ /path/to/root/style/
    Alias /js/ /path/to/root/js/

    Alias /admin/css/ /path/to/root/admin/css/
    Alias /admin/img/ /path/to/root/admin/img/
    Alias /admin/js/ /path/to/root/admin/js/

    <Directory /path/to/root/>
        Options -Indexes
        # backwards compatibility with apache 2.2
        Order allow,deny
        Allow from all

        # forward compatibility with apache 2.4
        Require all granted
        Satisfy Any
    </Directory>

    WSGIScriptAlias / /path/to/root/wsgi.py
    WSGIPassAuthorization On
    WSGIDaemonProcess <server name> processes=2 threads=15 display-name=%{GROUP}
    WSGIProcessGroup <server name>

    <Directory /usr/local/www/wsgi-scripts>
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

Everyday apache rotates its logs (using logrotate) and issues a greaceful restart (service reload).

However, Not all child processes exit. One is hanging and apache continues to send requests to it for processing, but that child is unable to run properly, constantly outputting the following error message to the old log file:

(2)No such file or directory: [client [my ip]:50202] mod_wsgi (pid=32714): Unable to connect to WSGI daemon process '[daemon process name]' on '/var/run/apache2/wsgi.2629.0.1.sock' as user with uid=33

This is causing the webserver to return 503s.

After a proper restart the issue is resolved for a day until the next log rotation.

As far as I understand, the children are supposed to finish whatever they are doing and exit, so I thought that maybe some keep alive is preventing that (some high frequency api calls are made to this server), so I tried adding KeepAlive off but it didn't work.

Anybody has any idea what is going on here? Thanks.

EDIT: As a workaround I have edited the apache logrotate file to do a restart instead of a reload post rotation. Now I get

child process 4591 still did not exit, sending a SIGTERM

and

child process 4591 still did not exit, sending a SIGKILL

I expected this to happen even for a reload after a process does not exit. Does apache have such an option?

may be you are looking for https://stackoverflow.com/questions/31567165/what-is-the-difference-between-apache2-reload-restart-graceful/45608106 — asktyagi, Sep 12 '19 at 09:43
I saw this post, I am just saying that it seems like reload should force subprocesses to die at some point, otherwise you can get to the situation I am in. My main issue here is still the fact that this subprocess does not want to die gracefully. — Yotam Alon, Sep 12 '19 at 11:02
Using mod_state? I'll try to get it during tomorrow's rotation. Do you think I should remove the workaround before hand? — Yotam Alon, Sep 15 '19 at 06:36
I meant was may be your application having persistent connection and no way to stop it gracefully, so what you should so better stop applications first gracefully and post that apache. — asktyagi, Sep 16 '19 at 03:33
I realize this is years too late, but did you figure this out? I have the exact same issue. My current best guess is that this is a possible candidate for the issue: https://bz.apache.org/bugzilla/show_bug.cgi?id=63169 — samGbos, Mar 11 '22 at 17:06
Just as a followup, I wrote more detailed answer here: https://serverfault.com/questions/849923/http-503-errors-with-apache-and-mod-wsgi — samGbos, Mar 16 '22 at 21:41

Apache child process does not exit and is not terminated

0 Answers0