12

I'm currently migrating some server software from Java 8 to Java 11. Everything works fine except for the fact that TLS connections are refused when my server runs on Java 11 (and only 11). I.e., everything works when running on up to and including Java 10. As soon as I switch to Java 11 (with absolutely no code changes inbetween, of course) I get a TLS decrypt error. My application works fine otherwise when I disable SSL/TLS or when I use IE/Edge to connect to my server (via TLS). When using Chrome 70 or Firefox 62 I get ERR_SSL_PROTOCOL_ERROR or SEC_ERROR_BAD_SIGNATURE, respectively.

I tried to analyze this using WireShark (see my dump). My conclusion is that Java 11 and Chrome/Firefox negotiate a bad cipher suite (rsa_pss_rsae_sha256), which somehow doesn't work with Java 11. Java 8 and IE/Edge seem to negotiate a different cipher suite (rsa_pkcs1_sha256), which works.

Does someone know how to fix this problem or at least work around it for the moment (other than disabling SSL/TLS or using IE/Edge, of course)? Thanks in advance.

yetanotheruser98345873498
  • 123
  • 5
  • 1
    Java 11 was released less than a month ago. Do you really want to run production on something this new? – Bart Friederichs Oct 16 '18 at 07:56
  • 5
    @BartFriederichs Who said I'm running this in production? I'm currently only in the process of migration. This is a step I'll have to do eventually. And I'd rather do it sooner than later, since Java 8 will be obsolete by january. – yetanotheruser98345873498 Oct 16 '18 at 08:10
  • I would test your server with CURL (uses AFAIK also NSS library like Firefox) and OpenSSL. Both tools allow to get more detailed error messages than Firefox. Also disabling TLS1.3 on Java side may be worth a try. – Robert Oct 16 '18 at 15:18
  • 1
    Have you tried an [SSL debug](https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html) option? If yes please paste the trace here. – Mikhail Kholodkov Oct 16 '18 at 20:22
  • Are you using OpenJDK or Oracle's JDK? I think that OpenJDK is missing some certificates. Maybe that's the reason –  Oct 19 '18 at 11:29
  • @Robert I already tried CURL and OpenSSL. Both have the same problem in that they negotiate a different cipher, which does not produce this error. – yetanotheruser98345873498 Oct 22 '18 at 06:12
  • @MikhailKholodkov Yeah, I've tried that already. All in all it only says I get a `SSLHandshakeException` with the message `decrypt error`. – yetanotheruser98345873498 Oct 22 '18 at 06:13
  • @a_horse_with_no_name I'm using Oracle JDK on Windows 7. – yetanotheruser98345873498 Oct 22 '18 at 06:14

1 Answers1

2

This is likely to be a compatibility failure with the new PSS algorithms in TLS 1.3. You can read a good explanation for the reasoning behind the introduction of PSS here.

Until it's fixed you can prevent your server from negotiating TLS 1.3 by editing lib/security/java.security and adding TLSv1.3 to the jdk.tls.disabledAlgorithms property.

Andy Brown
  • 11,766
  • 2
  • 42
  • 61
  • I thought as much. I also found that neither TLS 1.3 nor 1.2 are working for me. I have to disable both in Firefox to get it working. But thanks for your workaround, I'll try it out. – yetanotheruser98345873498 Oct 22 '18 at 05:58
  • 4
    I tried your solution. It doesn't work when I disable TLS 1.3 (and 1.2), because my JDK somehow falls back to TLS 1.0 and then proceeds to spit out a horrible amount of exceptions. I **did** manage to get it to work when I disabled RSA-PSS (by inserting all possible variations of `RSA(SSA)?[-_]PSS` into `jdk.tls.disabledAlgorithms`). So, thank you! – yetanotheruser98345873498 Oct 26 '18 at 07:49
  • 1
    @yetanotheruser98345873498 can you please paste here the entire jdk.tls.disabledAlgorithms property you used? Thnks! – sergiofbsilva Nov 16 '18 at 21:28
  • 2
    @sergiofbsilva You need to append `, RSASSA-PSS` to `jdk.jar.disabledAlgorithms` in your `${JAVA_HOME}/conf/security/java.security` to disable all broken signature schemes which use this algorithm in OpenJDK 11. I also found the corresponding bug report https://bugs.openjdk.java.net/browse/JDK-8216039. I also want to post the actual error message to help other people in finding this solution: "javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error". This error message can only be seen if `javax.net.debug` system property is enabled (I used `-Djavax.net.debug=all`). – Valentin Kovalenko Feb 12 '19 at 05:02
  • 1
    Hi all, does anyone found a solution on this issue with Java 11, I tried to add RSASSA-PSS to jdk.jar.disabledAlgorithms in conf/security/java.security but still have the issue – Djoz Mar 04 '19 at 08:49
  • Was hopeful but this does not fix it for me (openjdk-11.0.2, oraclejdk-11.0.2), nor did trying openjdk-12. :-( – cjstehno Mar 25 '19 at 13:52