1

Recently we started migrating our application from websphere-liberty 16.0.0.2 to version 17.0.0.2 (in both cases using javaee7 profile). With the same SSL configurations in server.xml the application is not able to invoke remote rest service over https due to SSL handshake failure. Here is my server.xml

<?xml version="1.0" encoding="UTF-8"?>
<server description="Default server">

    <featureManager>
        <feature>appSecurity-2.0</feature>
        <feature>transportSecurity-1.0</feature>
        <feature>jaxrs-2.0</feature>
        <feature>json-1.0</feature>
        <feature>javaMail-1.5</feature>
        <feature>ssl-1.0</feature>
    </featureManager>

    <sslDefault sslRef="defaultSSLSettings" />
    <ssl id="defaultSSLSettings" keyStoreRef="defaultKeyStore" trustStoreRef="defaultTrustStore" clientAuthentication="true" sslProtocol="TLSv1" />
    <keyStore id="defaultKeyStore" location="/opt/ibm/wlp/output/defaultServer/resources/security/key.jks" password="**********" />
    <keyStore id="defaultTrustStore" location="/opt/ibm/wlp/output/defaultServer/resources/security/trust.jks" password="***********" />

    <basicRegistry id="basic" realm="BasicRealm">
        <!-- <user name="yourUserName" password="" />  -->
    </basicRegistry>

    <variable name="defaultHostName" value="default-host.com" />

    <!-- To allow access to this server from a remote client host="*" has been added to the following element -->
    <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="9080" httpsPort="9443" />

    <!-- Automatically expand WAR files and EAR files -->
    <applicationManager autoExpand="true"/>

    <logging logDirectory="/var/log/wlp" traceSpecification="*=INFO:SSL=all" traceFileName="trace.log" consoleLogLevel="info" copySystemStreams="true"/>

</server>

The only difference in server.xml for websphere-liberty v.16 is that there is no transportSecurity-1.0 feature enabled.

The rest service that the application invokes over https provides SSL certificate signed by "GeoTrust Global CA". If import the certificate into the truststore 'trust.jks' then everything works fine but I expect the certificate to be automatically accepted during handshake since it's not self-signed.

What changes were done in websphere-liberty 17.0.0.2 related to SSL? Are there any additional security configurations should be done in server.xml?

Servers versions:

WebSphere Application Server 17.0.0.2 (1.0.17.cl170220170523-1818) on IBM J9 VM, version pxa6480sr4fp10-20170727_01 (SR4 FP10) (en_US)
WebSphere Application Server 16.0.0.2 (1.0.13.cl160220160526-2258) on IBM J9 VM, version pxa6480sr3fp10-20160720_02 (SR3 FP10) (en_US)

Error stack trace:

server_1  | [INFO    ] FFDC1015I: An FFDC Incident has been created: "java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
server_1  |     java.security.cert.CertPathValidatorException: The certificate issued by CN=GeoTrust Global CA, O=GeoTrust Inc., C=US is not trusted; internal cause is: 
server_1  |     java.security.cert.CertPathValidatorException: Certificate chaining error com.ibm.ws.ssl.core.WSX509TrustManager checkServerTrusted" at ffdc_17.08.09_14.50.00.0.log
server_1  | [ERROR   ] CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN CN=*.api.ibm.com, O=International Business Machines, L=Armonk, ST=New York, C=US was sent from the target host.  The signer might need to be added to local trust store /opt/ibm/wlp/output/defaultServer/resources/security/trust.jks, located in SSL configuration alias defaultSSLSettings.  The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
server_1  |     java.security.cert.CertPathValidatorException: The certificate issued by CN=GeoTrust Global CA, O=GeoTrust Inc., C=US is not trusted; internal cause is: 
server_1  |     java.security.cert.CertPathValidatorException: Certificate chaining error
server_1  | [WARNING ] Interceptor for {https://dev.api.ibm.com/scx/test}WebClient has thrown exception, unwinding now
server_1  | Could not send Message.
server_1  | [ERROR   ] 2017-08-09 14:50:00 ExceptionMapper:23 - javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1  | javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1  |     at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:632)
server_1  |     at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:608)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1105)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1042)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:895)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:863)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:426)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1554)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1549)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.get(WebClient.java:1469)
server_1  |     at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.get(InvocationBuilderImpl.java:80)
server_1  |     at com.ibm.si.saas.sbs.service.SBSApiService.getCustomer(SBSApiService.java:84)
server_1  |     at com.ibm.si.saas.sbs.resource.SbsEndpointApiResource.getConsoles(SbsEndpointApiResource.java:49)
server_1  |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
server_1  |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
server_1  |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
server_1  |     at java.lang.reflect.Method.invoke(Method.java:508)
server_1  |     at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:632)
server_1  |     at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:118)
server_1  |     at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
server_1  |     at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:252)
server_1  |     at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)
server_1  |     at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:423)
server_1  |     at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
server_1  |     at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:61)
server_1  |     at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:99)
server_1  |     at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
server_1  |     at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:124)
server_1  |     at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
server_1  |     at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:134)
server_1  |     at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:149)
server_1  |     at com.ibm.websphere.jaxrs.server.IBMRestServlet.doGet(IBMRestServlet.java:115)
server_1  |     at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
server_1  |     at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:99)
server_1  |     at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1290)
server_1  |     at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:778)
server_1  |     at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
server_1  |     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1161)
server_1  |     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4983)
server_1  |     at com.ibm.ws.webcontainer31.osgi.webapp.WebApp31.handleRequest(WebApp31.java:528)
server_1  |     at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:315)
server_1  |     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1025)
server_1  |     at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:280)
server_1  |     at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:967)
server_1  |     at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:359)
server_1  |     at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:318)
server_1  |     at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:471)
server_1  |     at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:405)
server_1  |     at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:285)
server_1  |     at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:256)
server_1  |     at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:174)
server_1  |     at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:83)
server_1  |     at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
server_1  |     at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
server_1  |     at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
server_1  |     at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
server_1  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
server_1  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
server_1  |     at java.lang.Thread.run(Thread.java:785)
server_1  | Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://dev.api.ibm.com/scx/test/customer/321321321: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1  |     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
server_1  |     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:83)
server_1  |     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
server_1  |     at java.lang.reflect.Constructor.newInstance(Constructor.java:437)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1385)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1369)
server_1  |     at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:653)
server_1  |     at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
server_1  |     at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
server_1  |     at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:704)
server_1  |     at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1041)
server_1  |     ... 55 more
server_1  | Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1  |     at com.ibm.jsse2.k.a(k.java:17)
server_1  |     at com.ibm.jsse2.at.a(at.java:851)
server_1  |     at com.ibm.jsse2.D.a(D.java:333)
server_1  |     at com.ibm.jsse2.D.a(D.java:113)
server_1  |     at com.ibm.jsse2.E.a(E.java:79)
server_1  |     at com.ibm.jsse2.E.a(E.java:107)
server_1  |     at com.ibm.jsse2.D.r(D.java:610)
server_1  |     at com.ibm.jsse2.D.a(D.java:372)
server_1  |     at com.ibm.jsse2.at.a(at.java:558)
server_1  |     at com.ibm.jsse2.at.i(at.java:73)
server_1  |     at com.ibm.jsse2.at.a(at.java:357)
server_1  |     at com.ibm.jsse2.at.startHandshake(at.java:723)
server_1  |     at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:215)
server_1  |     at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:34)
server_1  |     at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1561)
server_1  |     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489)
server_1  |     at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:491)
server_1  |     at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:80)
server_1  |     at org.apache.cxf.transport.http.URLConnectionHTTP
server_1  | Conduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:370)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1586)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1615)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1559)
server_1  |     at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1356)
server_1  |     ... 61 more
server_1  | Caused by: java.security.cert.CertificateException: PKIXCertPathBuilderImpl could not build a valid CertPath.
server_1  |     at com.ibm.ws.ssl.core.WSX509TrustManager.checkServerTrusted(WSX509TrustManager.java:322)
server_1  |     at com.ibm.jsse2.az.checkServerTrusted(az.java:78)
server_1  |     at com.ibm.jsse2.E.a(E.java:5)
server_1  |     ... 79 more

Here is output when trying to connect using openssl:

openssl s_client -CAfile /opt/ibm/java/jre/lib/security/cacerts -connect dev.api.ibm.com:443
Certificate chain
 0 s:/C=US/ST=New York/L=Armonk/O=International Business Machines/CN=*.api.ibm.com
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust SSL CA - G3
 1 s:/C=US/O=GeoTrust Inc./CN=GeoTrust SSL CA - G3
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2859 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 9509AB7194AB9DE0ABE219ECAE442C5D2C93145946FC2285CB9C4F5CCC81514F
    Session-ID-ctx: 
    Master-Key: 335D047C41E5FE75096E53C9CBACC7C1CC8F6254872599EFE93C7AD5935638181AAF9240656BD44A858723C38108BB31
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1502310152
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
Andy Guibert
  • 41,446
  • 8
  • 38
  • 61
Yuriy Bondaruk
  • 4,512
  • 2
  • 33
  • 49
  • I see that the IBM JDK you are using has also changed. Have you tried using the exact same JDK in each case to isolate the problem to going from Liberty 16->17? I suspect the issue may be that when you changed JDKs the GeoTrust Global CA is no longer a trusted root certificate. – Andy Guibert Aug 09 '17 at 18:30
  • I checked /opt/ibm/java/jre/lib/security/cacerts and it seems the root certificate is there. Moreover I could connect with openssl successfully using that cacert file – Yuriy Bondaruk Aug 09 '17 at 20:24

1 Answers1

1

The way you describe things it sound like when you were running Liberty 16.0.0.2 the JSSE SSL Context was being used in your scenario. If the cacerts file was being used for trust then it must have been. To be honest I thought jaxrs should have fallen back to the Liberty SSLContext if you do not provide a ssl reference in your code. I could be wrong about that and will have to investigate.

So at 17.0.0.2 and using transportSecurity-1.0 feature you are now using the a Liberty SSLContext. This is the correct behavior with transportSecurity-1.0, jaxrs will only fall back to the JSSE default SSLContext if there is no ssl configuration in the server.xml file. The transportSecurity-1.0 feature works like that to support the new outbound SSL features, outbound default, https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_config_ssl_outbound.html and outbound SSL Filter feature, https://www.ibm.com/support/knowledgecenter/en/was_beta_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_config_ssl_outbound_filter.html.

So if you use the transportSecurity-1.0 feature you will need to add the signer to you Liberty truststore, Liberty does not do any auto accepting of signers.

I will have to investigate what the behavior should be if the ssl-1.0 feature is used.

Alaine
  • 419
  • 2
  • 5
  • Hi Alaine. I just tried to use cacerts directly by specifying a truststore and pointing it to the cacerts file and it works. So it seems that in Liberty 16.0.0.2 cacerts file was used to check certificates. So I need to configure SSL to use a custom keystore and default truststore. – Yuriy Bondaruk Aug 10 '17 at 19:03
  • I tried to use ssl-1.0 feature only (without transportSecurity-1.0) in Liberty 17.0.0.2 but the result was the same (SSLHandshakeException). Everything works fine if use defaultTrustStore in ssl configuration. So i have to use it or generate a custom truststore and import certificates from cacerts. Thanks for your help! – Yuriy Bondaruk Aug 10 '17 at 19:14
  • Okay, it seems that in 16.0.0.2 in your scenario that a JSSE SSLContext is being used, Liberty does not load the cacerts file in it's SSLContext. Well not unless you configure cacerts as a keystore/truststore. I will investigate what should be done with the ssl-1.0 feature and 17.0.0.2. Like I said before if using transportSecurity-1.0 it is working as expected. – Alaine Aug 10 '17 at 20:06
  • Did additional investigation with outbound SSL config and here what I have: 1) Following configuration works well for outbound calls ` ` – Yuriy Bondaruk Aug 10 '17 at 20:40
  • 2) This fails (default trustStore for outbounds): – Yuriy Bondaruk Aug 10 '17 at 20:46
  • outboundSSLRef will only work if transportSecurity-1.0 feature is enabled. Do you have it enabled, the features are not included in your configuration example. – Alaine Aug 11 '17 at 02:56
  • Yes it is. transportSecurity-1.0 is enabled feature. 'The server installed the following features: [servlet-3.1, ssl-1.0, jndi-1.0, transportSecurity-1.0, json-1.0, distributedMap-1.0, appSecurity-2.0, jaxrs-2.0, javaMail-1.5, jaxrsClient-2.0]'. And here is what I see in logs: 'provider = "IBMJCE" Object type = com.ibm.ws.ssl.core.WSX509TrustManager tsCfgAlias = "saasTrustStore" tsFile = "/opt/ibm/wlp/output/defaultServer/resources/security/serverTruststore.jks" peerHost = "dev.api.ibm.com" config = class com.ibm.websphere.ssl.SSLConfig@dfd6747d' – Yuriy Bondaruk Aug 11 '17 at 11:43
  • I opened another thread for problem with SSL config for outbound connections: https://stackoverflow.com/questions/45636285/ssl-config-for-outbound-connections-doesnt-work-in-websphere-liberty-17-0-0-2 – Yuriy Bondaruk Aug 11 '17 at 13:22