HaProxy giving - 503 Service Unavailable

Question

I am using below configuring for performing loadbalancing for two Tomcat servers. And I configured HAProxy to perform SSL/TLS bridging/re-encryption.

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2 debug

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
    ssl-server-verify none

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  ft_main
default_backend bk_main
bind 192.168.192.175:443 ssl crt /home/testuser/Software/apache-tomcat-7.0.32/keystore/haproxy-cert.pem no-sslv3
mode http

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend bk_main
    balance     roundrobin
    retries 3
server srv01 192.168.192.173:8443 weight 1 maxconn 100 check no-sslv3 ssl verify none 
server srv01 192.168.192.174:8443 weight 1 maxconn 100 check no-sslv3 ssl verify none

When I am starting the service I am getting following error

Message from syslogd@localhost at Aug 17 17:06:12 ...
 haproxy[2593]: backend bk_main has no server available!

And I am observing following logs in /var/log/haproxy.log

Aug 17 16:52:07 localhost haproxy[2495]: backend bk_main has no server available!
Aug 17 16:52:25 localhost haproxy[2496]: 120.117.50.250:52088 [17/Aug/2015:16:52:25.084] ft_main~ bk_main/ 556/-1/-1/-1/556 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:26 localhost haproxy[2496]: 120.117.50.250:52089 [17/Aug/2015:16:52:25.336] ft_main~ bk_main/ 1044/-1/-1/-1/1044 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:27 localhost haproxy[2496]: 120.117.50.250:52090 [17/Aug/2015:16:52:27.371] ft_main~ bk_main/ 268/-1/-1/-1/268 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:28 localhost haproxy[2496]: 120.117.50.250:52091 [17/Aug/2015:16:52:27.623] ft_main~ bk_main/ 671/-1/-1/-1/671 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:29 localhost haproxy[2496]: 120.117.50.250:52092 [17/Aug/2015:16:52:29.395] ft_main~ bk_main/ 273/-1/-1/-1/273 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:30 localhost haproxy[2496]: 120.117.50.250:52093 [17/Aug/2015:16:52:29.646] ft_main~ bk_main/ 701/-1/-1/-1/701 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:52:51 localhost haproxy[2496]: 120.117.50.250:52094 [17/Aug/2015:16:52:51.268] ft_main~ bk_main/ 274/-1/-1/-1/274 503 212 - - SC-- 1/1/0/0/0 0/0 "GET /test/healcheck.jsp HTTP/1.1"
Aug 17 16:57:15 localhost haproxy[2496]: 192.168.192.168:54329 [17/Aug/2015:16:57:15.560] ft_main/1: SSL handshake failure
Aug 17 17:00:34 localhost haproxy[2538]: Proxy ft_main started.
Aug 17 17:00:34 localhost haproxy[2538]: Proxy bk_main started.
Aug 17 17:00:34 localhost haproxy[2538]: Server bk_main/srv01 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 30ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 17 17:00:34 localhost haproxy[2538]: backend bk_main has no server available!
Aug 17 17:06:12 localhost haproxy[2593]: Proxy ft_main started.
Aug 17 17:06:12 localhost haproxy[2593]: Proxy bk_main started.
Aug 17 17:06:12 localhost haproxy[2593]: Server bk_main/srv01 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 25ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 17 17:06:12 localhost haproxy[2593]: backend bk_main has no server available!

Based on the "SC" flags present in above flag my understand is that it failed to establish a server connection. So is it a handshake issue? I am using self signed certificate. And in server directive I mentioned "verify none" so what can be the issue?

I don't know which version of haproxy you are running but mine complains when I have to backend servers with the same name and no id. (Eg: srv1). Are you sure you mean to connect to the backend via ssl? The frontend is terminating ssl and then you are establishing another ssl from the load balancer to the backend server? Perhaps the relevant config section from the backend servers would help? — Daniel Widrick, Aug 21 '15 at 22:51
According the documentation, the health check is made through ssl when using ssl on the backend. Can you check on the logs of your backend server if there is responding correctly the `/test/healcheck.jsp` URL? Also, is this URI correct? Did you mean `/test/healthcheck.jsp`? — fgbreel, Aug 22 '15 at 02:45
Can you access the backend servers by visiting https://192.168.192.173:8443/ in a browser? — Luc van Donkersgoed, Aug 23 '15 at 12:00
In the `backend bk_main` section, you use the same label (`srv01`) _twice_, reusing it for the different addresses. If your second server fails the SSL connection, HAproxy may stop there, and not bother checking the _first_ server. — Castaglia, Dec 28 '16 at 22:48
As for the backend servers, you might test their SSL connections directly _e.g._ using `openssl s_client -connect 192.168.192.173:8443`, and see whether the backends require/support only SSLv3 (for example). — Castaglia, Dec 28 '16 at 22:58
It's recommended to provide a default backend to provide 404 error messages in case the regular backends don't respond. You'll get a 503 if nothing is available. Comment in video from a developer presentation: https://youtu.be/2riqRErga3E?t=633 — duct_tape_coder, Aug 13 '19 at 15:15

score 1 · Answer 1 · answered Mar 18 '18 at 16:12

Of course try the connection to backends yourself:

openssl s_client -connect 192.168.192.173:8443

This way you'll remove most of possible causes.

What if s_client works, but haproxy doesn't? On SELinux enforcing (for example CentOS 7 with default settings) exactly this happens and you need to explicitly allow haproxy to connect to any backend port:

setsebool -P haproxy_connect_any 1

score 0 · Answer 2 · answered Mar 08 '16 at 17:34

0

Check your /etc/hosts file and make sure that your instance name isn't the same as what you're proxying. Generally it sets the hostname of the server to 127.0.0.1

answered Mar 08 '16 at 17:34

Kladskull

1,255
5
15
32

HaProxy giving - 503 Service Unavailable

2 Answers2