Apache Webserver block frontend connection on reverse proxy error

Question

We have a problem with a relatively dumb loadbalancer that sits in front of an Apache Webserver. The Apache Webserver again is a reverse proxy for an application server.

The problem starts when the application goes down. This makes the Apache throw an error code.

The problem is that the loadblancer only considers a hard tcp error for removing a server from the pool. This means that error pages will go through the loadbalancer to the user instead of the server just being removed from the pool.

Is it possible to configure Apache to reject a tcp request on backend a backend error?

Can you share a bit more details about your Apache setup? Also, what happens when the application server "goes down" -- is it returning a 5XX error code, TCP socket timeout, etc.? Assuming a simple reverse proxy and a 5XX error code from the backend, this answer is probably all you need to do: https://serverfault.com/questions/266924/apache-reverse-proxy-error-page-when-proxied-server-is-down — Garrett, Jul 13 '19 at 18:12

score 2 · Answer 1 · answered Jul 09 '19 at 04:31

Haproxy can reject tcp connections based on dynamic conditions, including the number of servers available on the backend. The config keyword is tcp-request connection. (Link to docs below.)

So, maybe put haproxy in front of Apache (I know, I know! Three layers?!?!) or maybe use haproxy instead of Apache.

Might look like this (not tested):

frontend http-in
  bind *:80
  mode http
  tcp-request connection reject if nbsrv(appfarm) lt 1
  default_backend appfarm

backend appfarm
  server appsrv01 127.0.0.1:8888 check

I'd be happy to flesh this out a bit if you think you could use it. It's an interesting problem.

This is not the worst Idea. I would be sad though because I want to install haproxy as the main laodbalancer instead of the inferior solution we have. — Christopher Perrin, Jul 09 '19 at 09:47

score 0 · Answer 2 · answered Jul 07 '19 at 09:48

This will probably not work with apache httpd itself. The problem is that your failed application makes apache httpd generate an error. This error only happens when you actually request an URL from that application. Apache httpd needs the URL to decide wether this is an error or not and to get the URL from the client it needs the TCP connection.

What you can do as a workaround: You can write a rather simple script that reads the ErrorLog stream on stdin and then blocks requests from your loadbalancers (maybe iptables) once the errors start to happen. That script can then be configured as a sink of apache httpd's error log using the ErrorLog directive.

This would disable an apache httpd backend server, but it will never reenable it automatically.

I guess that would be a way. But not a very good one. I hate that loadbalancer. — Christopher Perrin, Jul 07 '19 at 20:48
Another idea/workaround could be to setup Apache with `mod_proxy_balancer` having a second (fallback) `member` pointing to another backend. — hargut, Jul 09 '19 at 04:42

Javier Ruiz · Answer 3 · 2019-07-15T17:47:08.297

I will move the logic to the load balancer, if it is not possible to replace the existent load balancer, install a small nginx in each Apache server to proxy the connection.

[User] - [Load Balancer] - [Nginx] - [Apache]

By using nginx as load balancer or reverse proxy you have access to the upstream responses (In your case, Apache), and you can make rules that evaluate the response code before replying to the end client:

$upstream_status keeps status code of the response obtained from the upstream server. Status codes of several responses are separated by commas and colons like addresses in the $upstream_addr variable. If a server cannot be selected, the variable keeps the 502 (Bad Gateway) status code.

Nginx example rules

Call load balancer API to remove node, if available:

if ($upstream_status != 200) {
    return 301 return 301 http://loadbalance/api/remove/node;
}

Call local script (please review security) to kill Apache and tell user to reload page (it will use another backend next time). It uses ngx_http_lua_module.

if ($upstream_status != 200) {
   content_by_lua_block {
     os.execute("/bin/killapache.sh")
   } 
   return 301 https://$host$request_uri;
}

That won't work. The problem is that the loadbalanger will not remove a failed host from the pool if it is not "TCP down". I can send error codes already but I actually don't want the user to see them. — Christopher Perrin, Jul 10 '19 at 14:10
I see, then I will suggest to write a daemon that checks http connections and kills Apache if the response is wrong. You could also use a Zabbix agent or other monitoring solution, to check responses and send an action on failure. The action will be to kill Apache. — Javier Ruiz, Jul 10 '19 at 15:40

Apache Webserver block frontend connection on reverse proxy error

3 Answers3