Debugging SOAP webservice calls using CXF generated stubs in Java gives SSL Exceptions for some people

Question

We have a project setup where we call external SOAP webservices using CXF generated stubs. The SOAP calls use certificate based security. This all works perfectly except when debugging the code (we use IntelliJ).

When running the code, it works. When debugging and running from breakpoint to breakpoint, it works. When debugging step by step and actively going into or stepping over methods that call the SOAP services, the following error occurs for some people (I redacted some class and methodnames):

javax.xml.ws.WebServiceException: Could not send Message.
    at org.apache.cxf.jaxws.JaxWsClientProxy.mapException(JaxWsClientProxy.java:183)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
    at com.sun.proxy.$Proxy91.getData(Unknown Source)
    at com.mycompany.batch.service.MyService.getData(MyService.java:763)
    at com.mycompany.proxy.MyProxy.getData(MyProxy.java:152)
    at com.mycompany.model.MyAction.updateDossier(MyAction.java:91)
    at com.mycompany.model.MyAction.process(MyAction.java:80)
    at com.mycompany.model.AbstractAction.process(AbstractAction.java:68)
    at com.mycompany.MyTestCase.main(MyTestCase.java:33)
Caused by: javax.net.ssl.SSLException: SSLException invoking https://someexternalurl.com/v6: readHandshakeRecord
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1402)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1386)
    at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
    at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:673)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:531)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:440)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:355)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
    at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140)
    ... 10 more
Caused by: javax.net.ssl.SSLException: readHandshakeRecord
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1072)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1356)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1331)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:241)
    at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:274)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1345)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1306)
    at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:307)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)
    at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1358)
    ... 22 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
    at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
    at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
    at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251)
    at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
    at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:749)
    at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
    at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
    ... 35 more

We are a team of multiple developers and about 50% encounter this issue while the rest doesn't and can't reproduce this at all.

Since it only happens when debugging it might be an issue with how IntelliJ does the actual debugging.

I found this thread: Server-side handling missing client connection (reset by peer)?

It mentions a client thread hanging the server but no further info is given.

Does anyone have an idea what the problem might be?

EDIT: We have tried to debug the same code side-by-side and compare all settings through debug information. Everything we checked (and we checked A LOT) was the exactly the same.

Can you get any information from the server(s) what error they think is occurring, if any? If not, I would get an external capture with wireshark tcpdump or similar and see if you are actually gettng RST and if so its sequence and timing _on the wire_ relative to other messages. For most Java SSL/TLS problems I suggest sysprop `javax.net.debug=ssl:handshake` which is nicely formatted, but not here where timing may matter and debugging frequently changes timing quite a bit. — dave_thompson_085, Oct 12 '21 at 03:31
It's impossible to get info from the remote servers. It's a 3rd party and it's government. I'll look into the sysprop. — B.T., Oct 12 '21 at 11:03
@dave_thompson_085 SSL debug didn't have much extra info javax.net.ssl|DEBUG|01|main|2021-10-12 14:07:10 CEST|CertificateVerify.java:743|Produced CertificateVerify handshake message ( "CertificateVerify": { "signature algorithm": rsa_pkcs1_sha256 "signature": { ... } } ) javax.net.ssl|ERROR|01|main|2021-10-12 14:07:10 CEST|TransportContext.java:313|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking ( "throwable" : { javax.net.ssl.SSLException: readHandshakeRecord ... Caused by: java.net.SocketException: Connection reset by peer: socket write error ... } ) — B.T., Oct 14 '21 at 07:11
The log in the JVM can't show when the RST and any send actually happened, only when code did read or write; that's why in _this_ case I recommended an external capture. In particular I would want to compare it to the time interval between receiving the server's first flight and actually sending (not just writing) the client's second flight on a successful handshake. — dave_thompson_085, Oct 15 '21 at 05:41

score 1 · Accepted Answer · answered Jan 24 '22 at 13:37

We did a lot of extra research, also using Wireshark and we saw the RST but could not find a solution.

Until we saw that the working machines used JDK 11.0.5 and the failing machines use JDK 11.0.2. Upgrading to 11.0.5+ (11.0.14) solved our problem. The specific fix or change in the JDK subversion remains unknown.

Debugging SOAP webservice calls using CXF generated stubs in Java gives SSL Exceptions for some people

1 Answers1