Spring WebServiceTemplate causing frequent intermittent java.net.SocketException: Connection reset

Question

We are seeing frequent intermittent java.net.SocketException: Connection reset errors in our application logs. I have been reading similar other questions but nowhere I could find a relevant solution to fix or a way to reproduce the error in local environment. Could someone please guide?

HTTPClient code at client side (all below timeouts are set as 60000)

    @PostConstruct
    public void init() {
        Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
        marshaller.setContextPaths(CONTEXT_PATH);
        setMarshaller(marshaller);
        setUnmarshaller(marshaller);

        PoolingHttpClientConnectionManager poolingConnManager = new PoolingHttpClientConnectionManager();
        poolingConnManager.setDefaultMaxPerRoute(10);
        RequestConfig config = RequestConfig.custom().setConnectTimeout(CONNECT_TIMEOUT)
                .setConnectionRequestTimeout(REQUEST_TIMEOUT)
                .setSocketTimeout(SOCKET_TIMEOUT).build();
        CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(poolingConnManager)
                .addInterceptorFirst(new HttpComponentsMessageSender.RemoveSoapHeadersInterceptor())
                .setDefaultRequestConfig(config)
                .build();
        HttpComponentsMessageSender httpComponentsMessageSender = new HttpComponentsMessageSender(httpClient);
        getWebServiceTemplate().setMessageSender(httpComponentsMessageSender);
        getWebServiceTemplate().setDefaultUri("serverURL");
    }

Connection Reset error in logs

java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.springframework.ws.transport.http.HttpComponentsConnection.onSendAfterWrite(HttpComponentsConnection.java:121)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:48)
at org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceTemplate.java:658)
at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:606)
at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:555)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:390)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:383)

I wouldn't be able to use wireshark (as suggested at some places), as it will need taking a lot of permissions on production machines. Is it something obvious we are missing here. This error never came before and has started recently. There is no relevant change in applications which could be doubted on. In order to reproduce locally, I tried putting Thread.sleep in the server application for more than 1 minute, but that didn't reproduce the Connection Reset error. We are using spring 3, httpclient 4.5.9 and deploying client application on tomcat 8.

I would start by taking note of the very first entry of this error. Rollback to the previous version and deploy in isolation. Then step by step checkout the commits to identify the problem. — Aniket Sahrawat, Feb 02 '21 at 13:07
Done with that activity already, with no luck. There is no major changes made in the application which could be doubted on. Any suggestions how may I reproduce it, because that would be half battle won I guess. — Pallavi Mistry, Feb 02 '21 at 13:30
set the timeouts to very low like ~10ms and mock very high-frequency requests in the integration test. — Aniket Sahrawat, Feb 02 '21 at 13:57
Done with that too in our dev environments, but it didnt produce Connection reset error. It gave other errors, like timeouts etc, but reset didnt happen. Thanks for the suggestion though. — Pallavi Mistry, Feb 04 '21 at 13:43

Spring WebServiceTemplate causing frequent intermittent java.net.SocketException: Connection reset

0 Answers0