1

I have a cert file that is working fine under one Open Liberty instance (version 20.0.0.2 under jdk-11.0.5+10-openj9) to make outbound calls, using the standard HttpsURLConnection, to a particular endpoint that uses cert-auth. On a separate server of the same version/JDK, I am creating a new app using the MicroProfile REST Client APIs (using @Asynchronous, if that matters) but calls are failing with handshake failure. I am using the same cert and the same keystore config (and have enabled the "ssl-1.0" feature in both servers)...

<keyStore id="defaultKeyStore" location="key.jks" password="changeit" type="jks"/>

I tried debugging using "-Djavax.net.debug=all" under both servers and found that the cert with my target alias is found and added as trusted certificates in both cases upon server startup....

SunX509KeyManagerImpl.java:164|found key for : my-alias (...)
X509TrustManagerImpl.java:79|adding as trusted certificates (...)
SSLContextImpl.java:115|trigger seeding of SecureRandom
SSLContextImpl.java:119|done seeding of SecureRandom

But, for the mpRestClient app, when I make a call to the endpoint in question it seems to spontaneously switch trust stores to the default JDK cacerts trust store...

TrustStoreManager.java:112|trustStore is: C:\Program Files\AdoptOpenJDK\jdk-11.0.5+10-openj9\lib\security\cacerts
....
TrustStoreManager.java:311|Reload the trust store
TrustStoreManager.java:318|Reload trust certs
TrustStoreManager.java:323|Reloaded 88 trust certs
X509TrustManagerImpl.java:79|adding as trusted certificates (...)
SSLContextImpl.java:115|trigger seeding of SecureRandom
SSLContextImpl.java:119|done seeding of SecureRandom

After a bunch of negotiation happens, it eventually comes to...

CertificateMessage.java:290|No X.509 certificate for client authentication, use empty Certificate message instead
CertificateMessage.java:321|Produced client Certificate handshake message (
  "Certificates": <empty list>
)
....
TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
  "throwable" : {
    javax.net.ssl.SSLException: readHandshakeRecord
    ....
  }
)

Of course, there's no cert found for client auth because the whole truststore got reloaded, wiping out the originally loaded truststore. This spontaneous switching of trust stores does not occur under the other server. The relevant, successful, behavior on that server is...

SunX509KeyManagerImpl.java:401|matching alias: my-alias
ServerHelloDone.java:151|Consuming ServerHelloDone handshake message (
  <empty>
)
CertificateMessage.java:321|Produced client Certificate handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 3769
SSLSocketOutputRecord.java:255|Raw write (...)
RSAClientKeyExchange.java:193|Produced RSA ClientKeyExchange handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 262
SSLSocketOutputRecord.java:255|Raw write (...)
CertificateVerify.java:743|Produced CertificateVerify handshake message (...)
SSLSocketOutputRecord.java:241|WRITE: TLS12 handshake, length = 264
SSLSocketOutputRecord.java:255|Raw write (...)
ChangeCipherSpec.java:115|Produced ChangeCipherSpec message
SSLSocketOutputRecord.java:225|Raw write (...)
Finished.java:398|Produced client Finished handshake message (...)
....
ChangeCipherSpec.java:149|Consuming ChangeCipherSpec message
....
SSLSocketInputRecord.java:249|READ: TLSv1.2 handshake, length = 64
SSLCipher.java:1329|Padded plaintext after DECRYPTION (...)
Finished.java:535|Consuming server Finished handshake message (...)
SSLSocketOutputRecord.java:309|WRITE: TLS12 application_data, length = 339
SSLCipher.java:1483|Padded plaintext before ENCRYPTION (...)
SSLSocketOutputRecord.java:323|Raw write (...)
...
SSLSocketInputRecord.java:249|READ: TLSv1.2 application_data, length = 544
SSLCipher.java:1329|Padded plaintext after DECRYPTION (...)
[...and then I get my decrypted response...]

Another difference between the two server setups is that the successful one is a standalone download of the javaee8 version of Open Liberty while the unsuccessful one is executed via Maven...

mvn liberty:dev -Ddebug=false -DskipTests=true

I don't know why that should make any difference, but something clearly is. The configs are set up nearly identically. I've been searching far and wide and trying everything I could think of to tinker with, but now I'm just at a loss for what could be making the difference. Hopefully someone notices something. The key is to get the server running the mpRestClient app to stop spontaneously switching trust stores to the JDK cacerts default. But nothing I've tried prevents that so far.

Jacob Kjome
  • 13
  • 4
  • Hi Jacob, can you post the messages.log for the working and failing cases? I'm particularly interested in the list of features that are enabled and whether there are any warnings/errors regarding features or bundles. My guess is that the failing case is missing some feature or other. – Andy McCright Feb 19 '20 at 21:04
  • Features for the working server, includes the full microProfile-3.0 feature: `A CWWKF0012I: The server installed the following features: [appSecurity-2.0, cdi-2.0, concurrent-1.0, distributedMap-1.0, ejbLite-3.2, javaMail-1.6, jaxb-2.2, jaxrs-2.1, jaxrsClient-2.1, jaxws-2.2, jdbc-4.2, jndi-1.0, jpa-2.2, jpaContainer-2.2, json-1.0, jsonb-1.0, jsonp-1.1, jwt-1.0, localConnector-1.0, microProfile-3.0, mpConfig-1.3, mpFaultTolerance-2.0, mpHealth-2.0, mpJwt-1.1, mpMetrics-2.0, mpOpenAPI-1.1, mpOpenTracing-1.3, mpRestClient-1.3, opentracing-1.3, restConnector-2.0, servlet-4.0, ssl-1.0].` – Jacob Kjome Feb 20 '20 at 04:11
  • Features for the failing server, using a more skinnied-down feature set (this time with transportSecurity-1.0, but also have tried using only ssl-1.0 with no major difference in behavior): `A CWWKF0012I: The server installed the following features: [cdi-2.0, jaxrs-2.1, jaxrsClient-2.1, jndi-1.0, jsonp-1.1, localConnector-1.0, mpConfig-1.3, mpRestClient-1.3, servlet-4.0, ssl-1.0, transportSecurity-1.0].` – Jacob Kjome Feb 20 '20 at 04:19
  • I did just noticed this in the failing server, though it doesn't make sense to me since I do provide the password in the keystore in file `configDropins/defaults/keystore.xml`, as you can see above: `I CWPKI0819I: The default keystore is not created because a password is not configured on the element, and the 'keystore_password' environment variable is not set.` – Jacob Kjome Feb 20 '20 at 04:31
  • Now I see why. A few lines later, it processes the config dropins, so that shouldn't matter: `A CWWKG0093A: Processing configuration drop-ins resource: [...path to...]\target\liberty\wlp\usr\servers\defaultServer\configDropins\defaults\keystore.xml` – Jacob Kjome Feb 20 '20 at 04:35
  • Now I'm seeing something else, which could be the culprit, though I'm not sure why it's occurring?: `I FFDC1015I: An FFDC Incident has been created: "java.lang.ClassNotFoundException: com.ibm.ws.jaxrs20.appsecurity.security.JaxRsSSLManager com.ibm.ws.jaxrs20.client.security.LibertyJaxRsClientSSLOutInterceptor 153" at ffdc_20.02.19_07.03.16.0.log` – Jacob Kjome Feb 20 '20 at 04:38
  • `Stack Dump = java.lang.ClassNotFoundException: com.ibm.ws.jaxrs20.appsecurity.security.JaxRsSSLManager at java.base/java.lang.Class.forNameImpl(Native Method) at java.base/java.lang.Class.forName(Class.java:340) at com.ibm.ws.jaxrs20.client.security.LibertyJaxRsClientSSLOutInterceptor.getSocketFactory(LibertyJaxRsClientSSLOutInterceptor.java:135) at com.ibm.ws.jaxrs20.client.security.LibertyJaxRsClientSSLOutInterceptor.handleMessage(LibertyJaxRsClientSSLOutInterceptor.java:74) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)` – Jacob Kjome Feb 20 '20 at 04:49
  • It looks like the CWPKI0819I is happening because our default Instance of the defaultKeyStore is being processed before the config reads in the keystore.xml file. The defaultKeyStore from your keystore.xml is what should be used. I have not seen this happen before. – Alaine Feb 20 '20 at 14:49
  • What accounts for the ClassNotFoundException of `com.ibm.ws.jaxrs20.appsecurity.security.JaxRsSSLManager`? – Jacob Kjome Feb 20 '20 at 17:10
  • Sorry for the slow reply - I was out at DevNexus last week. Ultimately, I think this is a bug, so I'd recommend that you open an issue at https://github.com/OpenLiberty/open-liberty/issues (please be sure to reference this page). I think you can work around this issue by adding in the `appSecurity-2.0` feature. It kinda seems like JAX-RS requires both the SSL and AppSecurity features in order to use a different key store. Please give that a try and let me know if it works. I still think it is a bug though - and we should fix it if possible. Thanks! – Andy McCright Feb 22 '20 at 20:05

1 Answers1

1

Ultimately, I think this is a bug, so I'd recommend that you open an issue at https://github.com/OpenLiberty/open-liberty/issues (please be sure to reference this page). I think you can work around this issue by adding in the appSecurity-2.0 feature. It seems like JAX-RS requires both the SSL and AppSecurity features in order to use a different key store. That explains the ClassNotFoundException - the JaxRsSSLManager class is only added to the framework's classpath if the appSecurity-2.0 (or 3.0) feature is installed.

Using the AppSecurity feature should resolve the issue, but I consider that to be a workaround. If you open an issue with OpenLiberty, we can try to get this working without the AppSecurity feature.

Hope this helps, Andy

Andy McCright
  • 1,273
  • 6
  • 8
  • Adding the `appSecurity-2.0' (or 3.0) feature appears to be a partial solution. My key file appears to be loaded early and reports `I Successfully loaded default keystore: [...path to...]/target/liberty/wlp/usr/servers/defaultServer/resources/security/key.jks of type: jks`. I also added a dependency in the pom.xml to ensure downloading of the appSecurity feature as a provided `esa` lib from `io.openliberty.features`. However, I continue to receive the identical ClassNotFoundException for `com.ibm.ws.jaxrs20.appsecurity.security.JaxRsSSLManager` and continue to receive a handshake failure. – Jacob Kjome Feb 25 '20 at 00:32
  • I tried adding the appSecurity-2.0 (also tried 3.0) feature alternatively to either keystore.xml or server.xml. Behavior was no different. In both cases, the resulting features were reported as `A CWWKF0012I: The server installed the following features: [appSecurity-2.0, cdi-2.0, distributedMap-1.0, jaxrs-2.1, jaxrsClient-2.1, jndi-1.0, jsonp-1.1, mpConfig-1.3, mpRestClient-1.3, servlet-4.0, ssl-1.0].` – Jacob Kjome Feb 25 '20 at 00:36
  • I made some headway on this and got by the handshake exception. Indeed, the `appSecurity-2.0` (or 3.0) feature is required to use my own keystore (as suggested). But the `ClassNotFoundException` for `JaxRsSSLManager` was still interfering with that functionality. It occurred to me that maybe the cause was using a distribution of Open Liberty that was too limited in its functionality. In my pom.xml, for the `liberty-maven-plugin`, I had been referencing `openliberty-runtime` in my `assemblyArtifact` config. I tried pointing to `openliberty-microProfile3` and TLS finally worked! However... – Jacob Kjome Feb 27 '20 at 18:47
  • ...I'm now encountering an issue with hostname verification failing... and being enabled by default. My local development cert works against the server I'm calling (and trust), but the domain doesn't match the cert. Based on the [Open Liberty 19.0.0.6 announcement](https://openliberty.io/blog/2019/06/21/microprofile-rest-client-19006.html#ssl), I was under the impression that this would be disabled by default when using `transportSecurity-1.0` or, at the very least, I could disable it by using `verifyHostname="false"` on the `defaultSSLConfig`. Yet, hostname verification is always applied. – Jacob Kjome Feb 27 '20 at 19:11
  • How do I disable hostname verification so I can get past the following exception? `Caused by: java.lang.RuntimeException: HostnameVerifier, socket reset for TTL at org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:98) at java.base/sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:640) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:581)` – Jacob Kjome Feb 27 '20 at 19:14
  • I tried downgrading to Open Liberty 19.0.0.5 - because I surmised it might not include the hostname verification behavior at all given the announcement of its introduction in 19.0.0.6 - but I continue to encounter the same hostname verification issue when using `mpRestClient-1.2` (or 1.3). Is there any workaround? Am I just missing something? – Jacob Kjome Feb 27 '20 at 19:24
  • That's good to know that the distribution matters. Still something that should probably be fixed in Liberty. I'll look into it more. Thanks for letting me know! AFAIK, Hostname verification is on by default, but you can disable it. Here is the doc: https://download.eclipse.org/microprofile/microprofile-rest-client-1.3.4/microprofile-rest-client-1.3.4.html#_hostname_verification And here is an example: https://github.com/eclipse/microprofile-rest-client/blob/1.3.X-service/tck/src/main/java/org/eclipse/microprofile/rest/client/tck/interfaces/ssl/ConfigurableHostnameVerifier.java HTH – Andy McCright Feb 27 '20 at 20:28
  • Thanks. I'll check out the doc to disable hostname verification. FYI, I just reported [JAX-RS requires both the SSL and AppSecurity features in order to use a different key store #11103](https://github.com/OpenLiberty/open-liberty/issues/11103) – Jacob Kjome Feb 27 '20 at 20:52
  • FYI, hostname verification disabling didn't work for me. I reverted to `openliberty-microProfile3` version `20.0.0.2`, along with using `mpRestClient-1.3`. I created a custom `javax.net.ssl.HostnameVerifier` that simply returns `true` from `public boolean verify(String hostname, SSLSession session)`. I configured it in my `microprofile-config.properties` using `myRestClient/mp-rest/hostnameVerifier=com.acme.ssl.SSLAffirmativeHostnameVerifier`. I find no change in behavior. Hostname verification continues to fail with the same stacktrace mentioned previously. What am I missing??? – Jacob Kjome Feb 27 '20 at 22:03
  • FYI, I also tried switching from a CDI-loaded mpRestClient to a programmatically built client. I applied`myRestClientBuilder.hostnameVerifier(new SSLAffirmativeHostnameVerifier())` and used that client to call the URL in question with zero behavioral changes. I continue to get the same stacktrace as above where the `org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier` continues to be invoked as the hostname verifier. It appears that the mpRestClient config for hostname verification is being totally ignored under Open Liberty. Is this yet another bug or am I missing something? – Jacob Kjome Feb 27 '20 at 23:18
  • That is strange - Open Liberty passes the TCK which uses the HostnameVerifier in the link I mentioned previously. Can you open another GitHub issue for this? Please include the full stack trace and a code sample if possible. Thanks. – Andy McCright Feb 27 '20 at 23:29
  • Ok, I'll open another issue. BTW, I tried setting `verifyHostname="true"` on the `defaultSSLConfig`, just to see what would happen, and got the error log message `CWPKI0824E: SSL HANDSHAKE FAILURE: Host name verification error while connecting to host [someserver.acme.com]. The host name used to access the server does not match the server certificate's SubjectDN or Subject Alternative Name information. The extended error message from the SSL handshake exception is: [No subject alternative DNS name matching someserver.acme.com found.].` – Jacob Kjome Feb 27 '20 at 23:43
  • FYI, I just reported [mpRestClient-1.3 ignoring hostnameVerifier configuration #11108](https://github.com/OpenLiberty/open-liberty/issues/11108) – Jacob Kjome Feb 28 '20 at 01:24
  • Thanks for opening the issue. Can you try setting verifyHostname=“false” to see if that works? – Andy McCright Feb 28 '20 at 02:44
  • Well, `false` is the [default](https://openliberty.io/docs/ref/config/ssl.html). In any case, I've also explicitly set it to `false` and I get the same effect as not setting it at all. Search for 'verifyHostname="false"' and you'll find my earlier comments. What is confusing to me is the contradiction between you stating that "hostname verification is on by default" and the [Open Liberty 19.0.0.6 announcement](https://openliberty.io/blog/2019/06/21/microprofile-rest-client-19006.html#ssl) stating it was a new feature that is disabled by default. There appear to be 2 verification levels.... – Jacob Kjome Feb 28 '20 at 03:06
  • One level is the server SSL config, where one can optionally set `verifyHostname=“true”` where hostname verification is controlled by WSX509TrustManager: `Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching someserver.acme.com found. at com.ibm.ws.ssl.core.WSX509TrustManager.processServerTrustError(WSX509TrustManager.java:938) at com.ibm.ws.ssl.core.WSX509TrustManager.checkServerTrusted(WSX509TrustManager.java:729) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625) ... 95 more`. – Jacob Kjome Feb 28 '20 at 03:17
  • A second level of hostname verification appears to be applied by the Apache CXF (the Impl for mpRestClients in Open Liberty) when `verifyHostname=“false”` (or isn't set at all) via `org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier`. So, while one can enable hostname verification at a higher level, if no earlier hostname verification is performed then it is, eventually, performed by Apache CXF. This begs the questions of what use is the SSL config when there is already fallback hostname verification and why mpRestClients don't respect the `hostnameVerifier` config? – Jacob Kjome Feb 28 '20 at 03:33
  • See my latest comment in [mpRestClient-1.3 ignoring hostnameVerifier configuration #11108](https://github.com/OpenLiberty/open-liberty/issues/11108), where I've shown that an older-style JAX-RS-2.1 client works for configuring a custom hostname verifier, thus proving that the Open Liberty `hostnameVerifier` functionality for mpRestClient-1.3 is broken rather than it being something specific about my environment/setup. – Jacob Kjome Feb 28 '20 at 06:20