7

I have tried searching for this, and I have found things that are close, but I have not found anything that helps so far. I apologize in advance if this is already asked.

I have a service which is behind an HTTP VIP, and that is working fine. Now I am trying to add a TCP VIP and do TLS termination on the host, but when I run cURL to test if it is working (please let me know if there is a better way to test the connection), I get the following error, but I do not know what it means. From what I have read from this link, I think it is something to do with the certificate step not working, but I do not know how to figure out why it is not working.

Does anyone know what I am doing wrong or how to go about troubleshooting this? If I need to provide any additional information, please let me know. I am not very experienced with networking, so a lot of this is new to me.

% curl -v https://my-tcp-vip.example.com/explorer/model.json
*   Trying 192.0.2.x...
* TCP_NODELAY set
* Connected to my-tcp-vip.example.com (192.0.2.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error

If it is of any use, I am using the following version of cURL.

% curl --version
curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

Update

After running openssl s_client, I get the following output:

openssl s_client -connect my-tcp-vip.example.com:443
CONNECTED(00000005)
4791412332:error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/ssl/ssl_pkt.c:1205:SSL alert number 80
4791412332:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1569522740
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

I am still not that familiar with networking. Does this mean there is something wrong with the certificate or pem file or key file or what else does this mean? I am sorry if this is a dumb question. I have just never really done much networking work before.

Update 2

Here is the openssl s_client command using -msg.

openssl s_client -connect my-tcp-vip.example.com:443 -msg
CONNECTED(00000005)
>>> TLS 1.2 Handshake [length 00c3], ClientHello
    01 00 00 bf 03 03 9f b4 25 72 5a d7 be aa 41 ba
    4c a6 e0 a9 88 13 98 86 09 a0 bb a6 67 69 95 aa
    44 4c ef 8a 21 86 00 00 60 c0 30 c0 2c c0 28 c0
    24 c0 14 c0 0a 00 9f 00 6b 00 39 cc a9 cc a8 cc
    aa ff 85 00 c4 00 88 00 81 00 9d 00 3d 00 35 00
    c0 00 84 c0 2f c0 2b c0 27 c0 23 c0 13 c0 09 00
    9e 00 67 00 33 00 be 00 45 00 9c 00 3c 00 2f 00
    ba 00 41 c0 11 c0 07 00 05 00 04 c0 12 c0 08 00
    16 00 0a 00 15 00 09 00 ff 01 00 00 36 00 0b 00
    02 01 00 00 0a 00 08 00 06 00 1d 00 17 00 18 00
    23 00 00 00 0d 00 1c 00 1a 06 01 06 03 ef ef 05
    01 05 03 04 01 04 03 ee ee ed ed 03 01 03 03 02
    01 02 03
<<< TLS 1.2 Alert [length 0002], fatal internal_error
    02 50
4598130284:error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/ssl/ssl_pkt.c:1205:SSL alert number 80
4598130284:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.260.1/libressl-2.6/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1569525554
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Thank you so much for your help.

ashipma
  • 171
  • 1
  • 1
  • 3
  • 1
    "please let me know if there is a better way to test the connection" You can use `openssl s_client` which has flags for verbose debug. Also look at https://github.com/noxxi/p5-ssl-tools/blob/master/analyze-ssl.pl and https://testssl.sh/ – Patrick Mevzek Sep 26 '19 at 15:44
  • "I think it is something to do with the certificate step not working" In that case you would have a more explicit error. "internal error" is quite vague. Also it happens right after "ServerHello" based on your dump, which is before the certificate handling part of the handshake. Do you control the server too? Have access to its logfiles? – Patrick Mevzek Sep 26 '19 at 15:45
  • "how to go about troubleshooting this?" If you gave the real names involved, people could have tried to connect too and maybe see the problem. And please do not use `mydomain.com` to obfuscate, as this is a real name and certainly not yours. Use guidance from RFC2606 which is TL;DR: use example.com and example.net as "fake" names. Same for IP addresses, `1.2.3.4` exists and is not yours. See RFC5737 and TL;DR: use 192.0.2.x as block of "fake" IPv4 addresses – Patrick Mevzek Sep 26 '19 at 15:45
  • Thanks so much for your comments and suggestions. I have updated the IP address and the domain as you suggested. I could give the real domains and IP but they are internal to the company I work for, so no one else would be able to access them anyway, that is why I tried to give placeholder values. I do not have access to the VIP, but I do have access to the server. But when I look at the server's application logs, I do not see anything, presumably because the request never made it to the application. Do you know where to look for this type of log? I will try openssl like you suggested. – ashipma Sep 26 '19 at 18:02
  • I have provided the update with the openssl s_client command output. Do you know what this means? It seems to be saying there is no certificate, but I am not sure how to troubleshoot that, if that is actually what it is saying. Thanks again. – ashipma Sep 26 '19 at 18:37
  • I found this [Stack overflow link](https://stackoverflow.com/questions/24457408/openssl-command-to-check-if-a-server-is-presenting-a-certificate) that seems to be having the same issue, but they just changed some of the parameters in the openssl command. But this is for a webservice that I am trying to get to work. Maybe no certificate is found, but I do have the pem, key, and jks files all on the server. – ashipma Sep 26 '19 at 18:57
  • The problem happens BEFORE the certificate exchange, so should be unrelated to that. Are you sure your service does HTTPS? Run` openssl s_client ` with `-msg` for full details on what is exchanged (or a network sniffer at the same time), I suspect the server is not really doing TLS at all... – Patrick Mevzek Sep 26 '19 at 19:02
  • I have added a comment with the -msg flag. I am not sure how to decipher what it means though. – ashipma Sep 26 '19 at 19:22
  • "I am not sure how to decipher what it means though. " Your client started the TLS handshake sending ClientHello as expected and the server immediately replied with a fatal error, which means there was something in the ClientHello record that it did not accept. Also try to always use `-servername`. You can try various `-ssl3`, `-tls1`, `-tls1_1`, etc... – Patrick Mevzek Sep 27 '19 at 00:30

2 Answers2

11

Adding the server name as follows worked for me:

openssl s_client -connect server.com:443 -servername server.com
estani
  • 2,151
  • 2
  • 17
  • 13
  • Why does this work? The manpage for _-servername_ says: "_Set the TLS SNI (Server Name Indication) extension in the ClientHello message to the given value. If -servername is not provided, the TLS SNI extension will be populated with the name given to -connect if it follows a DNS name format._" If the value for _-servername_ is the same as the _-connect_ value, should it be the same as omitting _-servername_? (ref: https://www.openssl.org/docs/manmaster/man1/openssl-s_client.html) – ozmo Aug 11 '22 at 01:31
  • Ah -- answered my own question. At the time of this post, the version of openssl that ships with MacOS is an old version of the utility that ships as part of "LibreSSL 2.8.3", and which doesn't pass the hostname as SNI when -servername is omitted. Recent versions of the real openssl do pass hostname as SNI by default. – ozmo Aug 11 '22 at 02:31
0

If possible please try to use same openssl version on both server and client, this will possibly resolve if there is any limitations in supported ciphers on either side.

Karthik
  • 114
  • 4
  • Unfortunately I do not have control over the openssl version installed. The version on the host is `OpenSSL 1.0.1k-fips 8 Jan 2015`. I have connected to the host using ssh and tried that command again, but it is still having the same error. – ashipma Sep 26 '19 at 19:48
  • what's the openssl version you have on the client machine? If it's new can you try downgrading openssl version to match with the old one in the server? – Karthik Sep 27 '19 at 07:52