haproxy ssl termination works on http BUT fails on https with 503 - on Virtualbox apache2 ubuntu 18.04 backend server

Question

I've just dived into haproxy, and my goal is to setup an ssl terminated loadbalancer with one frontend ubuntu, and (for now) one backend lamp (ubuntu). Both the frontend and backend machines are 18.04 ubuntu's on my virtualbox. I am on Windows 10, incase that matters

I've gone through tons of docs and answers for the last few days, but I still can't fix this.

These 2 work fine :

1. The backend server serves the site without problems if I access it via http from a browser ( http://mysite.testing )
2. The backend server serves the site without problems if I access the site directly via the ip address of the backend server (http://192.168.56.108)

Note : There is no ssl cert on the backend server. But I have setup a mkcert on the haproxy machine, which works fine - green lock n all

What doesnot work / The issue :

When I access the https site via haproxy ( https://mysite.testing ) the page shows me a 503 Service Unavailable - No server is available to handle this request

My setup :

I've added this line to my hosts file, which is how I access mysite.testing :

192.168.56.105 mysite.testing 

My haProxy.cfg file :

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode    http
    option forwardfor
    option http-server-close
    option  httplog
    option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend FrontEnd_Http
    bind *:80
    reqadd X-Forwarded-Proto:\ http
    default_backend My_Web_Servers

frontend FrontEnd_Https
    bind 192.168.56.105:443 ssl crt /etc/haproxy/ssl/mysite.testing.pem
    mode http
    #reqadd X-Forwarded-Proto:\ https
    default_backend My_Web_Servers


backend My_Web_Servers
    mode http
    balance roundrobin
    server tad108 192.168.56.108

listen stats
    bind :32700
    stats enable
    stats uri /
    stats auth admin:admin

My apache2 conf file ( /etc/apache2/sites-enabled ) :

<VirtualHost *:80>
        ServerName mysite.testing
        ServerAlias www.mysite.testing
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html/mysite.testing/public_html

        ErrorLog /var/www/html/mysite.testing/logs/error.log
        CustomLog /var/www/html/mysite.testing/logs/access.log combined

    <Directory />
                Options FollowSymLinks
                AllowOverride All
        Require all granted
    </Directory>

</VirtualHost>

Questions :

• What might be causing the 503 ?
• Is this a problem with the backend apache2 server or with my frontend haproxy.cfg ?
• How do I fix this, so as to get the site working with https://mysite.testing without 503?

Many thanks for reading :)

Answer 1

When you configure a server like this...

server tad108 192.168.56.108

...you haven't told HAProxy what port the server uses to accept traffic, so HAProxy assumes you intended a 1:1 equivalent mapping of the possible bind ports on the HAProxy frontend where the traffic arrived, and the target port on the backend, so traffic arriving on 80 goes to the server on 80, while traffic arriving on 443 goes to the server on 443. This is rarely what you want, which is typically to send all traffic to a single port, like this:

server tad108 192.168.56.108:80

The ability to preserve the initial port has some interesting possibilities for other applications, but would rarely be useful for HTTP and HTTPS since in mode http, the server is expected either to speak TLS or not, depending on whether the ssl option is configured.

You can also preserve the port with an offset.

<port> is an optional port specification. If set, all connections will be sent to this port. If unset, the same port the client connected to will be used. The port may also be prefixed by a "+" or a "-". In this case, the server's port will be determined by adding this value to the client's port.

http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server

(The "client's port," as used here, refers to the port the client connected to, not the source port on the client's TCP stack.)