When migrating a proprietary Java (and Jetty) based application from RHEL7 to RHEL8 I learned something new: At least with OpenJDK 11 the JVM still defaults to a 1024 bit Diffie-Hellman group unless one specifies the system property jdk.tls.ephemeralDHKeySize. Nobody noticed before since all client connections are handled via a Load Balancer but this was finally discovered because a Python client running on the same host wasn't able to connect anymore due to the DEFAULT system-wide cryptographic policy which requires at least 2048 bits.
While setting this system property solved the issue, digging into the badly documented logic around update-crypto-policies(8) showed some config entries which hint that this system property should be set already according to the policy.
At least in /usr/lib/jvm/java-11-openjdk/conf/security/java.security there exists a cryptic comment which refers to a file managed by that tool:
# Determines whether this properties file will be appended to
# using the system properties file stored at
# /etc/crypto-policies/back-ends/java.config
#
security.useSystemPropertiesFile=true
And that file is a symlink which indeed contains these system properties when the DEFAULT policy is chosen:
jdk.tls.ephemeralDHKeySize=2048
jdk.certpath.disabledAlgorithms=MD2, MD5, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=DH keySize < 2048, TLSv1.1, TLSv1, SSLv3, SSLv2, DHE_DSS, RSA_EXPORT, DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_DSS_EXPORT, DH_RSA_EXPORT, DH_anon, ECDH_anon, DH_RSA, DH_DSS, ECDH, 3DES_EDE_CBC, DES_CBC, RC4_40, RC4_128, DES40_CBC, RC2, HmacMD5
jdk.tls.legacyAlgorithms=
Oddly, for the LEGACY policy it tries to set a 1023 bit key size:
jdk.tls.ephemeralDHKeySize=1023
jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1023
jdk.tls.disabledAlgorithms=DH keySize < 1023, SSLv3, SSLv2, RSA_EXPORT, DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_DSS_EXPORT, DH_RSA_EXPORT, DH_anon, ECDH_anon, DH_RSA, DH_DSS, ECDH, DES_CBC, RC4_40, DES40_CBC, RC2, HmacMD5
jdk.tls.legacyAlgorithms=3DES_EDE_CBC, RC4_128
Whichever policy is set, the server will always use 1024 bits which I verified via sleep 1 | openssl s_client -tls1_2 -cipher EDH -connect rhel8.example:443 | grep "Server Temp Key"
I didn't find any Jetty related issue which indicates that this wouldn't work there but didn't check yet if this affects these proprietary applications only. But setting that property explicitly via -D works so I don't see why setting those properties indirectly shouldn't work either. (It would probably also work to add an explicit -Djava.security.properties=/etc/crypto-policies/back-ends/java.config to the command line.)
I also couldn't find any official documentation on security.useSystemPropertiesFile.
So how and in which context is this supposed to work anyway? And is there maybe some documentation somewhere which I missed?
