HAProxy backend server returns "SSL handshake error"

Question

I know it's a frequently asked question which often means there's a problem with certificate validation. It doesn't seem to be the case, because I do not verify the certificate.

This is how my server specification looked in the beginning:

server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl verify none check resolvers mydns

Later it evolved to

server 1.base.maps.ls.hereapi.com ipv4@1.base.maps.ls.hereapi.com:443 ssl verify none force-tlsv12 check resolvers mydns resolve-prefer ipv4

But it always returns the same error:

Server freehere_maps_redirect/1.base.maps.ls.hereapi.com is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure"

URL is the real server name. There are 3 more servers in this backend (2.base.maps, 3.base.maps, 4.base.maps...), and all of them return the same error. These servers are not under my control.

I also tried to enable certificate verification, but, of course, it didn't fix the problem. No need to say that curl and other attempts to connect to the backend server work flawlessly.

HA-Proxy version 1.8.4-1deb90d 2018/02/08 running on CentOS7.

Your ideas will be greatly appreciated :)

Update: Here is the output of a successful request made with curl:

$ curl -v --tls-max 1.2 'https://1.base.maps.ls.hereapi.com/maptile/2.1/maptile/newest/normal.day/11/525/761/256/png8?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

*   Trying 13.33.243.105:443...
*   Trying 2600:9000:2118:be00:2:b190:a500:93a1:443...
* Connected to 1.base.maps.ls.hereapi.com (13.33.243.105) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=NL; ST=Noord-Brabant; L=Eindhoven; O=HERE Global BV; CN=*.base.maps.ls.hereapi.com
*  start date: May 10 17:54:34 2020 GMT
*  expire date: May 11 17:54:34 2021 GMT
*  subjectAltName: host "1.base.maps.ls.hereapi.com" matched cert's "*.base.maps.ls.hereapi.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b593a60020)
> GET /maptile/2.1/maptile/newest/normal.day/11/525/761/256/png8?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/2
> Host: 1.base.maps.ls.hereapi.com
> user-agent: curl/7.72.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< content-type: image/png
< content-length: 30653
< access-control-allow-origin: *
< cache-control: public,max-age=86400
< date: Thu, 03 Dec 2020 12:51:37 GMT
< etag: 4742ccdf6d
< expires: Fri, 04 Dec 2020 12:51:37 GMT
< last-modified: Thu, 19 Nov 2020 20:58:37 GMT
< server: nginx
< x-nlp-irt: D=68945
< x-ols-tid: hjBM-ZZ2dSrOCS0R3IovQk_icd0LanCeb81VKszvSQzzTLygjksosg==
< x-served-by: i-094a2ddaf9791ccf9.eu-west-1b
< x-cache: Hit from cloudfront
< via: 1.1 228e9f9ffd3a938a52da99b2c67d587f.cloudfront.net (CloudFront)
< x-amz-cf-pop: HEL50-C1
< x-amz-cf-id: gALOyDPx1QY7dsP0NlaEBivgFHjgP8kLWPw5l2U2DfS1ZCEXOg7FDQ==
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* stopped the pause stream!
* Connection #0 to host 1.base.maps.ls.hereapi.com left intact

Update2: I tried to replace HTTP checks with TCP checks by modifying the backend to:

backend freehere_maps_redirect
        http-send-name-header Host
        http-request set-uri http://%[req.hdr(Host)]%[path]?apiKey=xxxxxxxxxxx&%[query]
        option tcp-check
        tcp-check connect port 443
        server 1.base.maps.ls.hereapi.com ipv4@1.base.maps.ls.hereapi.com:443 check

tcp-check passes, but I still can't get any results from the uplink server. On the other hand, it may point to a deeper problem, because I get HTTP 400 (Bad request) from CloudFront:

<H1>400 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Bad request.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Update 3: Some more details. I tried to use ssldump to view the SSL handshake process. Here's how it looks with haproxy:

1 1  0.1478 (0.1478)  C>S  Handshake
      ClientHello
        Version 3.3 
        cipher suites
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        TLS_DH_DSS_WITH_AES_256_GCM_SHA384
        TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
        TLS_DH_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
        TLS_DH_RSA_WITH_AES_256_CBC_SHA256
        TLS_DH_DSS_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DH_RSA_WITH_AES_256_CBC_SHA
        TLS_DH_DSS_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA
        TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_DH_DSS_WITH_CAMELLIA_256_CBC_SHA
        TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
        TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_GCM_SHA384
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        TLS_DH_DSS_WITH_AES_128_GCM_SHA256
        TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
        TLS_DH_RSA_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
        TLS_DH_RSA_WITH_AES_128_CBC_SHA256
        TLS_DH_DSS_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DH_RSA_WITH_AES_128_CBC_SHA
        TLS_DH_DSS_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_SEED_CBC_SHA
        TLS_DHE_DSS_WITH_SEED_CBC_SHA
        TLS_DH_RSA_WITH_SEED_CBC_SHA
        TLS_DH_DSS_WITH_SEED_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA
        TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_DH_DSS_WITH_CAMELLIA_128_CBC_SHA
        TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
        TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_GCM_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_SEED_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_IDEA_CBC_SHA
        TLS_ECDHE_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
        TLS_ECDH_RSA_WITH_RC4_128_SHA
        TLS_ECDH_ECDSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
1 2  0.3019 (0.1541)  S>C  Alert
    level           fatal
    value           handshake_failure
1 3  0.3019 (0.0000)  S>C  Alert
    level           warning
    value           close_notify
1    0.3024 (0.0004)  S>C  TCP FIN
1    0.3112 (0.0087)  C>S  TCP FIN

And here's the successful exchange using curl:

New TCP connection #1: localhost.localdomain(60630) <-> server-13-33-243-73.hel50.r.cloudfront.net(443)
1 1  0.2425 (0.2425)  C>S  Handshake
      ClientHello
        Version 3.3 
        cipher suites
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xcca9
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xcca8
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xccaa
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_AES_256_GCM_SHA384
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_AES_128_GCM_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        compression methods
                  NULL
1 2  0.3793 (0.1367)  S>C  Handshake
      ServerHello
        Version 3.3 
        session_id[32]=
          f8 f2 8b 5b 48 eb bb 7f d8 5c 70 e0 9c 86 30 0d 
          f7 3d 6c 52 2f 66 b7 33 84 09 1f bb 25 14 d9 f6 
        cipherSuite         TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        compressionMethod                   NULL
1 3  0.3994 (0.0201)  S>C  Handshake
      Certificate
1 4  0.3994 (0.0000)  S>C  Handshake
      ServerKeyExchange
1 5    0.3994   (0.0000)    S>C    Handshake
        ServerHelloDone
etc.

Does it mean it's the ciphers mismatch problem? But the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 selected by the remote server during curl exchange is also present in the ciphers list offered by haproxy.

Tried also to add sni str(d2t4vhngii5504.cloudfront.net) or sni str(1.base.maps.ls.hereapi.com), but it didn't help either.

Are the certs actually valid? Are you sure your HAProxy server and the backend servers have compatible TLS settings (Ciphers etc)? — jstuart-tech, Dec 03 '20 at 12:23
you might be failing due to the check, i would test it forcing a tcp-check using the `option tcp-check` on the backend configuration and explicitly declaring the port to be checked using `check port 443` — BANJOSA, Dec 03 '20 at 12:26
*"... often means there's a problem with certificate validation."* - SSL handshake error is usually unrelated to validation of the server certificate. It is more related to problems with a client certificate, no shared ciphers, unsupported TLS version or missing or wrong SNI. It might be useful to add the successful `curl -v` output to your question in order to better understand the problem. Same is true about details for "other attempts", whatever these are. — Steffen Ullrich, Dec 03 '20 at 12:46
@jstuart-tech, yes, the certificates are valid. They are issued by GlobalSign nv-sa and signed by GlobalSign Root CA - R3. It is possible that the root CA is missing from my store, but I do not verify the certificate — minaev, Dec 03 '20 at 12:48
@SteffenUllrich , I have added the output of curl using TLS1.2 to the question. — minaev, Dec 03 '20 at 12:59
@BANJOSA, I tried to replace HTTP checks with TCP. Results added to the question. Pretty curious. — minaev, Dec 03 '20 at 13:24

Steffen Ullrich · Accepted Answer · 2023-06-13T13:40:54.487

4

server ... ipv4@1.base.maps.ls.hereapi.com:443 ssl ... check ...

The server in question requires SNI. Thus check-sni name instead of check need to be used. From the documentation:

check-sni
This option allows you to specify the SNI to be used when doing health checks over SSL. It is only possible to use a string to set . If you want to set a SNI for proxied traffic, see "sni".

Therefore it should be:

server ... ipv4@1.base.maps.ls.hereapi.com:443 ssl ...  check-sni 1.base.maps.ls.hereapi.com

edited Jun 13 '23 at 13:40

answered Dec 15 '20 at 19:41

Steffen Ullrich

13,227
27
39

Sorry, it didn't work either. My attempt was: ```server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl sni str(1.base.maps.ls.hereapi.com) verify none check check-sni str(1.base.maps.ls.hereapi.com)``` Handshake failure – minaev Dec 17 '20 at 07:12
@minaev: the `str(...)` is wrong, just put the name directly. See the updates response. – Steffen Ullrich Dec 18 '20 at 14:48
Looks like you have finally solved the riddle. I don't know as yet whether resolvers will work in this configuration, but this seems to work: `server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl verify none resolvers hetznerdns resolve-prefer ipv4 check-sni 1.base.maps.ls.hereapi.com check`. Thank you, Steffen! – minaev Dec 18 '20 at 16:35
"instead of `check`" - I supplied both and my server kept getting marked as DOWN due to handshake failures. Using only `check-sni` works flawlessly. Thanks! – Etienne Bruines Jun 13 '23 at 13:28
@EtienneBruines: Thanks for pointing out the error in the recommended solution. It should not have been both `check` and `check-sni` as initially shown, but `check-sni` instead of `check` as also described earlier in the answer. I've fixed it. – Steffen Ullrich Jun 13 '23 at 13:42

HAProxy backend server returns "SSL handshake error"

1 Answers1