nginx proxy retry while backend is restarting

Question

Nginx returns 502 while the backend is restarting. How can I make nginx proxy retry N times with M seconds delay? Is there a plugin or something else?

Answer 1

I found that Nginx took ~2 seconds to exhaust attempts if you specified hundreds of instances of the same backend:

    server  localhost:8080 max_fails=0;
    server  localhost:8080 max_fails=0;
    server  localhost:8080 max_fails=0;
    server  localhost:8080 max_fails=0;

(.... er, repeat as required!)

Yes, a horrendous kludge - but it does add a degree of tenacity...

Even worse than that, you could use:

    server  localhost:8080 max_fails=0;     
    server  localhost:80 backup;        

Assuming Nginx is running on port 80, this would try to continually loop the request around Nginx until localhost:8080 responds. I.e. retry an infinite(?) number of times with zero seconds delay.

I'll get back to my padded cell now....

Answer 2

If your nginx installation has Lua support then you can hold on the client for some time with sleep method. The operation is non-blocking and doesn't lock the worker. Keep in mind that user can't be held infinitely as some other network related socket/firewall timeouts may occur finally.

server {
    listen 8502;
    location / {
        #25 seconds sleep
        content_by_lua_block {
             ngx.sleep(25); 
             ngx.exit(ngx.HTTP_BAD_GATEWAY);
        }
    }
}

Then in your upstream list you need to add above server as backup to hold a client.

upstream backend {
    server 127.0.0.1:3001 fail_timeout=2s; #The backend
    server 127.0.0.1:8502 backup; #Lua holding server in the event backend is restarting
}

And this should be included in your proxied location specification:

proxy_read_timeout         30;  #Value must be higher than sleep in Lua
proxy_next_upstream error timeout http_502 http_504;

Answer 3

Closest thing I know is this project, but I never used. I know that varnish has backend checking, maybe you want to take a look on it.