11

I have my own server (where I'm running Apache/2.4.27), and today I realized that from (Brave and Google Chrome - different computers) I'm getting from my websites this error;

This site can’t provide a secure connection

mywebsite.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

And the strange thing is that I'm getting this error every fifth click on my website.

From my conf file:

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/mywebsite/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mywebsite/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/mywebsite/chain.pem
SSLCompression off

from options-ssl-apache.conf;

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLHonorCipherOrder     on
SSLCompression          off

I have checked log file from website but nothing, also nothing here; /var/log/apache2/error.log

I'm trying to figure out what is causing this error, any ideas where can I find more info or even better, how to solve this problem?

EDIT:

If I try openssl s_client -connect mywebsite.com:443, it will return:

I'm using: OpenSSL 1.1.0f

CONNECTED(00000003)

...

3073276480:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:469:

ANOTHER EDIT:

As @quadruplebucky suggested I changed options-ssl-apache.conf into:

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite           HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1
SSLHonorCipherOrder     on
SSLCompression          off

#SSLSessionTickets       off

I also tried to add SSLProtocol all -SSLv2 -SSLv3 into my virtualhost conf file, and in a same time I did change couple of things here; /etc/apache2/mods-available/ssl.conf

#SSLCipherSuite HIGH:!aNULL
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1

SSLHonorCipherOrder on

#   The protocols to enable.
#   Available values: all, SSLv3, TLSv1, TLSv1.1, TLSv1.2
#   SSL v2  is no longer supported
SSLProtocol all -SSLv2 -SSLv3

EDIT:

After changing LogLevel into Info it returns:

[Sat Jul 08 13:34:53.374307 2017] [ssl:info] [pid 8710] [client] AH02008: SSL library error 1 in handshake (server mywebsite:443)
[Sat Jul 08 13:34:53.374717 2017] [ssl:info] [pid 8710] SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message
[Sat Jul 08 13:34:53.374750 2017] [ssl:info] [pid 8710] [client] AH01998: Connection closed to child 1 with abortive shutdown (server mywebsite:443)

EDIT:

If I run with option -crlf, like this: 

openssl s_client -crlf -connect mywebsite:443

I'm getting no error?

One more thing if I change LogLevel to debug, before that error I'm getting this:

[Tue Jul 11 23:00:38.641568 2017] [core:debug] [pid 26561] protocol.c(1273): [client 188.64.25.162:23165] AH00566: request failed: malformed request line
[Tue Jul 11 23:00:38.641634 2017] [headers:debug] [pid 26561] mod_headers.c(900): AH01503: headers: ap_headers_error_filter()

So after this that same error will happen:

SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message

openssl version
OpenSSL 1.1.0f  25 May 2017
user134969
  • 439
  • 2
  • 6
  • 18
  • It shouldn't be possible for _any_ endpoint config error to cause 'bad decrypt or mac' (alert 20). Is there anything in the network _between_ these clients and this server, like a firewall, IDS, IPS, DLP, or even a 'smart' router? Can you test repeatedly (since this may be data dependent and thus quasi-random) connecting from the server itself, or from a machine on the same segment as the server? – dave_thompson_085 Jul 09 '17 at 05:24
  • show the relevant contents of the *.443 virtualhost (the whole virtualhost if needed), also the output of "apachectl -S" so we can discard misconfiguration. – Daniel Ferradal Jul 10 '17 at 06:31
  • error message complaints about SSL v3, yet in your config it's disabled... – alexus Jul 11 '17 at 21:08
  • You say you are using OpenSSL 1.1.0f. Is that on *both* the server and on the machine where you issued the s_client commands above? What platform is your server running on? You might want to see if the error occurs with specific ciphersuites, E.g. what happens if you add "-cipher AES128-SHA" onto the end of your s_client command? – Matt Caswell Jul 13 '17 at 10:56
  • ninjaed: @alexus: _function_ and _file_ names and some literals `ssl3*` and `SSL3*` in OpenSSL are also used for TLS (1.0 through 1.2) because of the technical similarities between those protocols. user134969: 'length too short' also should _never_ be caused by any config. If that's repeatable please try to get a `s_client -debug` (with plain-RSA `-cipher` if you didn't do that on the server) and wireshark capture _for the same event_ so we can look at the actual wire data and compare it to what the program sees. – dave_thompson_085 Jul 15 '17 at 12:25

5 Answers5

9

You can't tell from that error if your server is negotiating SSLv3 or TLSv1 (you might want to have a look at this question on Unix & Linux and make sure it's disabled everywhere in apache...) --- the 1.1.0f source code here on GitHub deliberately blurs the two... 

   if (enc_err < 0) {
    /*
     * A separate 'decryption_failed' alert was introduced with TLS 1.0,
     * SSL 3.0 only has 'bad_record_mac'.  But unless a decryption
     * failure is directly visible from the ciphertext anyway, we should
     * not reveal which kind of error occurred -- this might become
     * visible to an attacker (e.g. via a logfile)
     */
    al = SSL_AD_BAD_RECORD_MAC;
    SSLerr(SSL_F_SSL3_GET_RECORD,
           SSL_R_DECRYPTION_FAILED_OR_BAD_RECORD_MAC);
    goto f_err;
}

So you might want to reorder your cipher suite:

SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1

This post on askubuntu about the POODLE vulnerability has an excellent list of resources for SSL inspection and plumbing.

The Mozilla config generator is an excellent public service.

The "getting this error every fifth click" comment is a little strange. Do you mean clicks, or every fifth line is a bad request in the logs? Try starting apache single-threaded (the -X flag) and see if that does the same thing...or maybe setting SSLSessionTickets off.

My thinking here is to eliminate threading and session/cache coherency/consistency as source of trouble. Running apache single-threaded (starting it with the -X flag) is one way to accomplish this, another way is to set MaxClients=1 (at least with the MPM model). Session tickets have been a source of trouble in the past with TLSv1.2 and they are enabled by default, this is the reasoning behind SSLSessionTickets off (note this is part of the SSL "Server Hello" message, not a session cookie or similar). The 'every fifth click' error still bothers me - I can't help but notice that most browsers will pipeline four resource requests in a single one...) and open a new connection (a new ssl handshake, etc...) for the fifth...without a packet capture it's hard to say what's actually going on.

It would seem that you've eliminated cipher negotiation as a source of error (you can duplicate the error condition under much more restrictive cipher specs, unless I'm mistaken). I would be curious to know if you can trigger the error by renegotiating SSL (just for kicks, say): openssl s_client -connect server:443 and then type 'R', see what the logs say.
Also see if session caching is working with the -reconnect option to s_client.
Something must be different about the receiving contexts for the SSL requests, and it seems the best way to figure that out (short of a byte-by-byte inspection of what's going over the wire, which might be tough to anonymize) is to severely limit the size of what's listening (that is, the number of listeners).

Other debugging tools I'd try (assuming posting packet captures is out of the question....)
- ssltap (in libnss3-tools on ubuntu)
- cipherscan
- sslscan

UPDATE
Poking at this through ssltap looks an awful lot like OpenSSL bug #3712 resurfaced (key renegotiation during read/write, basically). Looking for a decent workaround that won't kill performance. Fun stuff!

quadruplebucky
  • 5,139
  • 20
  • 23
  • 1
    I'd also play with more explicit SSLCipherSuite declarations (along the lines of what the Mozilla config generates), set LogLevel to info or debug, tcpdump/wireshark pcaps, etcetera, anything to figure what's actually happening. – quadruplebucky Jul 08 '17 at 03:41
  • 1
    Why don't you try a tool like cipherscan https://github.com/mozilla/cipherscan or sslscan https://github.com/rbsec/sslscan/ (also in many repos) to try to figure out what's actually going on before regressing? – quadruplebucky Jul 08 '17 at 20:55
  • With `SSLProtocol -SSLv3` the connection definitely wasn't SSL3. Remember in OpenSSL functions (and sourcefiles) named with ssl3 are still part of all TLS protocols up to 1.2. Deleting SSLv3 (and TLSv1 in 1.1.0 only) _ciphers_ actually prevents any _protocol_ below TLS1.2 being negotiated, but with up-to-date browsers that should be okay. – dave_thompson_085 Jul 09 '17 at 05:11
  • @user134969: for wireshark to decrypt these connections (and thus be of help here) you need to (temporarily) restrict the keyexchange to plain-RSA; this is easiest done on the Apache side with `SSLCiphers RSA+AES`. (And provide the server privatekey file in Edit/Preferences/Protocols/SSL.) Chrome used to complain about plain-RSA (i.e. _not_ PFS) but I just retested and mine seems to be happy now. – dave_thompson_085 Jul 09 '17 at 05:25
  • @quadruplebucky can you point me what is the location of "ssl3_record.c " in the system? – user134969 Jul 12 '17 at 18:52
  • It's part of the source code to OpenSSL, you would not normally have it on a system. The source tree path is in my link: ssl/record/ssl_record.c – quadruplebucky Jul 14 '17 at 17:32
  • The problem with your config is that for reasons I don't understand well OpenSSL does a really lousy job of mutiplexing the TLS connection and when a cipher renegotiation happens in the middle of a read or write it (which is fine, usually) fouls up the tls_record code in your case - something to do with the extended server secret TLS extension code, I think - that's the piece I can't exactly duplicate. – quadruplebucky Jul 15 '17 at 23:34
6

I've seen this before, had this before in fact.

The answer in my case ended up being extremely subtle.

The network adapter enabled TCP Segment Offloading, which due to a bug of some form was mangling (or truncating, cant remember) the last few bytes of some messages -- which was subsequently causing the MAC on the SSL records to fail.

It was a virtual machine on VMWare.

I would try disabling TSO/GSO/GRO in your environment and seeing if the problem goes away.

ethtool -K eth0 tso off gro off gso off ufo off
Matthew Ife
  • 23,357
  • 3
  • 55
  • 72
3

There is some weirdness with OpenSSL and multithreading. What MPM do you use? If this is multithreading-related the "prefork" should be safe while "worker" and "event" might be affected.

If your load-profile allows it maybe you can try to switch over to prefork and see if the issue persists.

Andreas Rogge
  • 2,853
  • 11
  • 24
1

Having similar error sometimes when I work with my smartphone as a wifi hubspot. Rebooting the connexion solve the problem.

Dizuite
  • 11
  • 2
0

First make sure chrome is updated. Older versions give problems with certain ciphers. Had this issue with chromium myself with common sites like amazon, etc.

Second, the advice to forbid protocols in "cipher list" you followed is a very bad idea because it won't forbid protocols, it will forbid most ciphers including those that work "since" SSLv3 (but does not mean you are enabling SSL3 if you allow SSLv3 ciphers), use a more generioc list provided by Mozilla SSL config generator for compatibility (note SSLv2 no longer exists or is supported in httpd or openssl so no reason to forbid it explicitately any more) and perhaps your previous combination was too strict, try this one:

SSLProtocol             all -SSLv3
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

If you still have issues, enable debugging for SSL and see what httpd has to say (only send 1 try or two and disable this logging, this is too noisy):

LogLevel ssl:trace3

Sidenotes: You can also go ahead and remove SSLCertificateChainFile since that directive is deprecated in 2.4. You can add the chain the SSLCertificateFile and sort all the certificates from leaf to root, or even change SSLCertificateChainFile to SSLCACertificateFile (although this one is mainly used for SSL auth).

You should also add (if you haven't yet) to add:

SSLRandomSeed startup file:/dev/urandom 2048
SSLRandomSeed connect file:/dev/urandom 2048
SSLSessionCache shmcb:/path/to/log/ssl_gcache_data(512000)

Edit: Following our conversation let's resort to check openssl installation and/or if it really is the same version your httpd uses:

Issue this command and let's see what it says: openssl ciphers -v 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'

Also to make sure the openssl version is the correct one run this:

openssl version
Daniel Ferradal
  • 2,415
  • 1
  • 8
  • 13