mod_proxy returns 503 errors even after proxied service is back up

Question

I have a setup with Apache2 as a front-end server for multiple python apps served by gunicorn. My Apache2 setup using mod_proxy looks like this:

<VirtualHost *:80>
    ServerName example.com
    UseCanonicalName On
    ServerAdmin webmaster@localhost

    LogLevel warn
    CustomLog /var/log/apache2/example.com/access.log combined
    ErrorLog /var/log/apache2/example.com/error.log
    ServerSignature On

    Alias /media/ /home/example/example.com/pysrc/project/media/

    ProxyPass /media/ !
    ProxyPass / http://127.0.0.1:4711/
    ProxyPassReverse / http://127.0.0.1:4711/
    ProxyPreserveHost On
    ProxyErrorOverride Off
</VirtualHost>

Generally, this setup works pretty well. I have one problem though: When I restart the gunicorn process (takes 2-5 seconds) and there is a request from Apache, that request will fail with a 503 error. So far so good. But Apache keeps returning 503 errors, even after the gunicorn process is back up. It resumes serving content from the proxied server only after a full restart of Apache.

Is there a way around this behaviour?

score 26 · Accepted Answer · edited Jan 02 '13 at 11:12

26

Add retry=0 to your ProxyPass lines:

ProxyPass / http://127.0.0.1:4711/ retry=0

From the mod_proxy documentation:

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

edited Jan 02 '13 at 11:12

Vebjorn Ljosa

662
1
5
13

answered May 07 '11 at 18:29

TimS

2,166
13
8

Thank you so much! Works perfectly :) You'll get the bounty in a couple of hours when I'm allowed to award it. – Benjamin Wohlwend May 07 '11 at 21:47
I believe you now want [this link](http://httpd.apache.org/docs/2.2/mod/mod_proxy.html) instead - the quoted paragraph seems to be missing from the version linked in the answer, and no mention of the `retry` option is made. For others wondering the same thing: the default value for this parameter is 60 seconds. – Gijs Nov 02 '12 at 14:14

score -1 · Answer 2 · answered May 07 '11 at 06:46

-1

Are you following the documented method for restarting gunicorn?

I would recommend a simple approach. If 2-5 seconds is acceptable downtime in your environment, then might I suggest simply scripting the Apache service to restart immediately after you restart your gunicorn service?

In a production environment, I would suggest using HAProxy instead of Apache as your front-end, and you might have much better luck.

answered May 07 '11 at 06:46

Matt Beckman

1,502
18
33

Thanks for your answer! I use [supervisord](http://supervisord.org/) to control my gunicorn processes, my `supervisor.conf` looks basically like the example in the [gunicorn repo](https://github.com/benoitc/gunicorn/blob/master/examples/supervisor.conf). Unfortunately, I can't move away from Apache as a front-end server for organizational reasons. – Benjamin Wohlwend May 07 '11 at 14:19

mod_proxy returns 503 errors even after proxied service is back up

2 Answers2