RabbitMQ client SSL handshake issue on JDK 11

Question

We have a RabbitMQ client running and started getting the following error on SSL handshake once we switched to JDK 11:

Caused by: java.io.EOFException: SSL peer shut down incorrectly

Our environment is:

Rabbitmq java client version: 5.7.3
OpenJDK 11.0.4
MacOS - 10.14.6

We have been running tests and getting a constant fail with EOF exception.
There are no changes on the client side code for working and not working tests. The only change is different server endpoints.
Both rabbitmq broker endpoints work for JDK version 8, 9 and 10.
I also added public certs of the broker to the client JDK keystore too.

Below is the stack trace of our tests:

javax.net.ssl|DEBUG|01|ScalaTest-run-running-RabbitMqConsumerTest|2019-11-07 13:57:41.429 GMT|ClientHello.java:653|Produced ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "3E 14 77 54 AB A2 BF 3D E0 56 F8 85 60 32 77 35 35 AC CB 71 D9 14 19 D9 0F 7C 93 C6 F7 DF 68 9D",
  "session id"          : "",
  "cipher suites"       : "[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030), TLS_RSA_WITH_AES_256_GCM_SHA384(0x009D), TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384(0xC02E), TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384(0xC032), TLS_DHE_RSA_WITH_AES_256_GCM_SHA384(0x009F), TLS_DHE_DSS_WITH_AES_256_GCM_SHA384(0x00A3), TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(0xC02F), TLS_RSA_WITH_AES_128_GCM_SHA256(0x009C), TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256(0xC02D), TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256(0xC031), TLS_DHE_RSA_WITH_AES_128_GCM_SHA256(0x009E), TLS_DHE_DSS_WITH_AES_128_GCM_SHA256(0x00A2), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384(0xC024), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384(0xC028), TLS_RSA_WITH_AES_256_CBC_SHA256(0x003D), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384(0xC026), TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384(0xC02A), TLS_DHE_RSA_WITH_AES_256_CBC_SHA256(0x006B), TLS_DHE_DSS_WITH_AES_256_CBC_SHA256(0x006A), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA(0xC00A), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA(0xC014), TLS_RSA_WITH_AES_256_CBC_SHA(0x0035), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA(0xC005), TLS_ECDH_RSA_WITH_AES_256_CBC_SHA(0xC00F), TLS_DHE_RSA_WITH_AES_256_CBC_SHA(0x0039), TLS_DHE_DSS_WITH_AES_256_CBC_SHA(0x0038), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256(0xC023), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(0xC027), TLS_RSA_WITH_AES_128_CBC_SHA256(0x003C), TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256(0xC025), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256(0xC029), TLS_DHE_RSA_WITH_AES_128_CBC_SHA256(0x0067), TLS_DHE_DSS_WITH_AES_128_CBC_SHA256(0x0040), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA(0xC009), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA(0xC013), TLS_RSA_WITH_AES_128_CBC_SHA(0x002F), TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA(0xC004), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA(0xC00E), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
  "compression methods" : "00",
  "extensions"          : [
    "status_request (5)": {
      "certificate status type": ocsp
      "OCSP status request": {
        "responder_id": <empty>
        "request extensions": {
          <empty>
        }
      }
    },
    "supported_groups (10)": {
      "versions": [secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
    },
    "ec_point_formats (11)": {
      "formats": [uncompressed]
    },
    "signature_algorithms (13)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp512r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "signature_algorithms_cert (50)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp512r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "status_request_v2 (17)": {
      "cert status request": {
        "certificate status type": ocsp_multi
        "OCSP status request": {
          "responder_id": <empty>
          "request extensions": {
            <empty>
          }
        }
      }
    },
    "extended_master_secret (23)": {
      <empty>
    },
    "supported_versions (43)": {
      "versions": [TLSv1.2, TLSv1.1, TLSv1]
    }
  ]
}
)
javax.net.ssl|ERROR|01|ScalaTest-run-running-RabbitMqConsumerTest|2019-11-07 13:57:41.724 GMT|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
    at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1322)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1160)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
    at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
    at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
    at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
    at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:123)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:160)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:170)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:305)
    at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:64)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1106)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1063)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1021)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1182)
    at generator.RabbitMQ$.getMQConnection(RabbitMQ.scala:58)
    at generator.RabbitMqConsumerTest.$anonfun$new$2(RabbitMqConsumerTest.scala:50)
  Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
    ... 80 more}
)
Working error logs:
javax.net.ssl|DEBUG|01|ScalaTest-run-running-RabbitMqConsumerTest|2019-11-07 13:32:26.482 GMT|ClientHello.java:653|Produced ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "58 45 2D 14 80 18 D7 96 51 01 8A 87 6C 74 5F D3 D9 A7 2B F8 4C EF 9C D7 0E EB 77 0F 99 1D 79 15",
  "session id"          : "",
  "cipher suites"       : "[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030), TLS_RSA_WITH_AES_256_GCM_SHA384(0x009D), TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384(0xC02E), TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384(0xC032), TLS_DHE_RSA_WITH_AES_256_GCM_SHA384(0x009F), TLS_DHE_DSS_WITH_AES_256_GCM_SHA384(0x00A3), TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(0xC02F), TLS_RSA_WITH_AES_128_GCM_SHA256(0x009C), TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256(0xC02D), TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256(0xC031), TLS_DHE_RSA_WITH_AES_128_GCM_SHA256(0x009E), TLS_DHE_DSS_WITH_AES_128_GCM_SHA256(0x00A2), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384(0xC024), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384(0xC028), TLS_RSA_WITH_AES_256_CBC_SHA256(0x003D), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384(0xC026), TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384(0xC02A), TLS_DHE_RSA_WITH_AES_256_CBC_SHA256(0x006B), TLS_DHE_DSS_WITH_AES_256_CBC_SHA256(0x006A), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA(0xC00A), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA(0xC014), TLS_RSA_WITH_AES_256_CBC_SHA(0x0035), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA(0xC005), TLS_ECDH_RSA_WITH_AES_256_CBC_SHA(0xC00F), TLS_DHE_RSA_WITH_AES_256_CBC_SHA(0x0039), TLS_DHE_DSS_WITH_AES_256_CBC_SHA(0x0038), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256(0xC023), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(0xC027), TLS_RSA_WITH_AES_128_CBC_SHA256(0x003C), TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256(0xC025), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256(0xC029), TLS_DHE_RSA_WITH_AES_128_CBC_SHA256(0x0067), TLS_DHE_DSS_WITH_AES_128_CBC_SHA256(0x0040), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA(0xC009), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA(0xC013), TLS_RSA_WITH_AES_128_CBC_SHA(0x002F), TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA(0xC004), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA(0xC00E), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
  "compression methods" : "00",
  "extensions"          : [
    "status_request (5)": {
      "certificate status type": ocsp
      "OCSP status request": {
        "responder_id": <empty>
        "request extensions": {
          <empty>
        }
      }
    },
    "supported_groups (10)": {
      "versions": [secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
    },
    "ec_point_formats (11)": {
      "formats": [uncompressed]
    },
    "signature_algorithms (13)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp512r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "signature_algorithms_cert (50)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp512r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "status_request_v2 (17)": {
      "cert status request": {
        "certificate status type": ocsp_multi
        "OCSP status request": {
          "responder_id": <empty>
          "request extensions": {
            <empty>
          }
        }
      }
    },
    "extended_master_secret (23)": {
      <empty>
    },
    "supported_versions (43)": {
      "versions": [TLSv1.2, TLSv1.1, TLSv1]
    }
  ]
}
)
javax.net.ssl|DEBUG|01|ScalaTest-run-running-RabbitMqConsumerTest|2019-11-07 13:32:36.811 GMT|ServerHello.java:871|Consuming ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "5D C4 1C EA CB 34 AD BF FE E1 91 89 40 4C 04 7A FF F1 A7 D2 0F C6 92 C6 F3 7B 20 C0 84 C2 B7 24",
  "session id"          : "32 E8 61 CF D7 89 E7 47 EC 00 2C A2 4D 78 86 07 6B 8E E8 B3 AC C5 41 E3 89 B8 DE 9C 93 A9 3A 5E",
  "cipher suite"        : "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030)",
  "compression methods" : "00",
  "extensions"          : [
    "renegotiation_info (65,281)": {
      "renegotiated connection": [<no renegotiated connection>]
    },
    "ec_point_formats (11)": {
      "formats": [uncompressed]
    }
  ]
}
)

rabbitmq server log -

2019-11-07 17:59:32.639 [info] <0.25302.1> TLS server: In state certify at ssl_connection.erl:757 generated SERVER ALERT: Fatal - Handshake Failure

Please provide RabbitMQ log entries from the same time period. — Luke Bakken, Nov 08 '19 at 11:11
"The only change is different server endpoints." - I am assuming by "server endpoint" you mean RabbitMQ servers. What is different between these servers? Are they running the same operating system and Erlang version? — Luke Bakken, Nov 10 '19 at 14:01
@LukeBakken they are both the same, using older JDK version it connected successfully but with JDK 11 we are having these problems with the same type of server implementation — Alex, Nov 10 '19 at 22:45
Are you, by any chance, using client certificate authentication? It smells like a client cert verification/missing error. — Izzy, Nov 12 '19 at 05:26
You should try enabling all cipher suites in the RabbitMQ configuration - https://www.rabbitmq.com/ssl.html#cipher-suites. Also, you should troubleshoot using [`openssl s_client`](https://www.rabbitmq.com/troubleshooting-ssl.html). Post the complete output of the command as a gist or as an attachment to a message to the [`rabbitmq-users`](https://groups.google.com/forum/#!forum/rabbitmq-users) list. Finally, if you provide code to reproduce this issue on that list I can run it to figure out what is up. Stack Overflow is a terrible forum for diagnosis. — Luke Bakken, Nov 12 '19 at 15:06
@Izzy's suggestion is a good one. If client cert auth fails the TLS handshake won't be completed. — Luke Bakken, Nov 12 '19 at 15:06
If you don't want to use google groups, please use our shiny new GH repo that is dedicated to discussions about RabbitMQ - https://github.com/rabbitmq/discussions — Luke Bakken, Nov 12 '19 at 15:07
Which version of Erlang are you using? I reported [an issue](https://bugs.erlang.org/browse/ERL-973) in the first releases of Erlang 22.0.x that would show up only on Java 11+. This may not be the cause of your problem, but it'd be good to exclude this possibility. As suggested by Luke, it'd be great to have some code and steps to reproduce to further investigate. — Arnaud Cogoluègnes, Nov 12 '19 at 15:13
Hello @Alex, have you tried with this.? use below line of code in your client program and try. System.setProperty("https.protocols", "TLSv1.2"); — Seshu Kandimalla, Nov 15 '19 at 16:24
@LukeBakken We cannot share the complete output of the s_client command, however here are last few lines of the output, which is a valid output as per the document https://www.rabbitmq.com/troubleshooting-ssl.html - ........ PSK identity: None PSK identity hint: None Start Time: 1574009554 Timeout : 300 (sec) Verify return code: 0 (ok) --- DONE ........ — Alex, Nov 17 '19 at 21:14
@ArnaudCogoluègnes The servers are 3rd party and they do not wish to disclose the exact version, however they have confirmed it is not version 22 — Alex, Nov 17 '19 at 21:14
Maybe this issue exists with other Erlang versions, but was never confirmed. There's really not enough information for us to work with at this point. — Luke Bakken, Nov 18 '19 at 18:32

score 1 · Answer 1 · answered Nov 11 '19 at 14:51

1

One of the features included in JDK 11 is the implementation of TLSv1.3.

See JEP 332 and JDK 11 features. More details in this issue.

In the stack trace of your tests there are TLSv1.2, TLSv1.1, TLSv1 as supported versions and TLSv1.2 as server and client versions, this is quite natural since today's RabbitMQ TLS supported versions are 1.1 and 1.2. (See docs). What surprises is that there should be a

All of this is a hypothesis based on the logs so I have for you a couple of things you can try:

Try adding the follow config and re run your tests. This in order to verify that the issue is indeed TLS compatibility related.

SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setExcludeProtocols("TLSv1.3");

Enable SSL handshake debug at Java with -Djavax.net.debug=ssl:handshake:verbose
Verify that your algorithms and ciphers are not eliminated in TLSv1.3.

If you have new insights about the issue, let me know, maybe I can help further :)

answered Nov 11 '19 at 14:51

imTachu

3,759
4
29
56

SslContextFactory you are referring to comes from the package Jetty, whereas the one which RabbitMQ client requires comes from the package - com.rabbitmq.client. The RabbitMQ client SslContextFactory does not have setExcludeProtocols method available. However, we did try the following 4 java start up arguments: -Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager -Dcom.sun.net.ssl.enableECC=false -Djsse.enableSNIExtension=false -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 – Alex Nov 11 '19 at 16:07
The log output you see above comes from the SSL debug start up command. It can be confirmed the client is using TLSv1.2, from the ClientHello message. – Alex Nov 11 '19 at 16:07
Sorry about the jetty suggestion :P Will try to find something else. In the meantime, can you please check the recommendations in https://www.rabbitmq.com/troubleshooting-ssl.html – imTachu Nov 11 '19 at 16:46
we looked through that documentation and tried all the recomendations but nothing helped – Alex Nov 11 '19 at 16:56

score 1 · Answer 2 · answered Feb 04 '20 at 10:12

I am not sure what is your issue because it isn't enough data to determine it, but it looks like a problem that I have.

I think that your problem started at jdk 11.0.3, so if you try 11.0.2 must work...

In my case:

The problem is that the invocation of ClientAuthManager.chooseClientAlias() was changed with JDK 11.0.3.

In JDK 11.0.2:

choseClientAlias({"EC","RSA","DSA"}, socket);

In JDK 11.0.3:

for (String keyType : {"EC","RSA","DSA"}) { 
    choseClientAlias({ keyType }, socket);
}

So when you have to test the valid certificates, only will do for the first call:

choseClientAlias({"EC"}, socket);

Maybe you could debug it at choseClientAlias, change the first keyType by "RSA" or "DSA", and prove if this is your issue.

RabbitMQ client SSL handshake issue on JDK 11

2 Answers2