Why browser is always showing certificate of only one server even if multiple certificates are configured in HAProxy?

Question

I am a beginner in HAProxy. I have four VMs, with HAProxy in one and Apache httpd in other three. What I am trying to achieve is, when user connect to HAProxy IP using https, the connection should be redirected to any of the other three servers. Please find my configuration below;

global
    maxconn 50000
    log /dev/log local0
    log /dev/log local1 notice
    user root
    group root
    stats timeout 30s
    nbproc 2
    cpu-map auto:1/1-4 0-3
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    daemon

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend ft_http
    bind :80
    mode http
    default_backend bk_http

frontend ft_https
    bind :443 ssl crt-list /etc/haproxy/crt-list.txt
    mode tcp
    default_backend bk_https

backend bk_http
    mode http
    balance roundrobin
    default-server inter 1s
    server testserver1 192.168.0.1:80 check
    server testserver2 192.168.0.2:80 check
    server testserver3 192.168.0.3:80 check

backend bk_https
    mode tcp
    balance roundrobin
    stick-table type ip size 200k expire 1m
    default-server inter 1s
    server testserver1 192.168.0.1:443 check
    server testserver2 192.168.0.2:443 check
    server testserver3 192.168.0.3:443 check

and below is how crt-list.txt looks like;

/etc/haproxy/testserver1.pem testserver1
/etc/haproxy/testserver2.pem testserver2
/etc/haproxy/testserver3.pem testserver3

I have generated certificates by referring these urls https://www.techrepublic.com/article/how-to-enable-https-on-apache-centos/ and https://www.suse.com/support/kb/doc/?id=000018152

Now, whenever I hit https://haproxy_ip in my browser, the browser prompt unsigned certificate and it always belongs to testserver1. Certificate of other two servers are not getting picked up.

And, When I accept the certificate and proceed, I am getting a page with content like below;

# Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

Apache/2.2.34 (Amazon) Server at testserver1.localdomain Port 443

and whenever I reload the page, the last line of message changes like Apache/2.2.34 (Amazon) Server at testserver2.localdomain Port 443 and Apache/2.2.34 (Amazon) Server at testserver2.localdomain Port 443 (but when I take the certificate, it is the one I created for testserver1)

Now, my question is, why am I always getting the certificate of testserver1?

Thanks in advance.

Answer 1

There is no load balancing in the certificates HAproxy offers.

The front-end always presents the best certificate for the hostname used in the SNI handshake. The best certificate is the one where the CN or a SAN entry in one of the certificates matches the URI used in the request and if no such match can be made, the default certificate is used.

The option crt-list uses the first entry as the default certificate.

Also the certificate presented at the front-end bears no relation to which back-end server HAProxy connects.
Unless your load balancer operates at the TCP/IP level and only forwards the connection, the load balancer is the man in the middle; the client connects to the load balancer and the load balancer makes its own new connection to the backend to forward the request

From the manual:

The first declared certificate of a bind line is used as the default certificate, either from crt or crt-list option, which haproxy should use in the TLS handshake if no other certificate matches. This certificate will also be used if the provided SNI matches its CN or SAN, even if a matching SNI filter is found on any crt-list. The SNI filter !* can be used after the first declared certificate to not include its CN and SAN in the SNI tree, so it will never match except if no other certificate matches. This way the first declared certificate act as a fallback