Spring RestTemplate SocketException Connection reset on quick consecutive executions

Question

Setup

Simple server and client applications running locally. Server endpoint receives a POST request with a sleep time to simulate work. Client is a SpringBoot app using RestTemplate for HTTP calls. Simulating a 500ms delay at the server per request and issuing 700 multi-threaded requests.

Problem

On quick consecutive executions of the client program, I get java.net.SocketException: Connection reset. Stack trace:

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8081/server/site": Connection reset; nested exception is java.net.SocketException: Connection reset

java.util.concurrent.ExecutionException: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8081/server/site": Connection reset; nested exception is java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_74]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_74]
at com.sample.client.rest.RestClient.invokeServer(RestClient.java:75) ~[classes/:na]
at com.sample.client.SampleClientApplication.main(SampleClientApplication.java:13) [classes/:na]

Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:8081/server/site": Connection reset; nested exception is java.net.SocketException: Connection reset
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:673) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:620) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:414) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at com.sample.client.rest.RestClient.lambda$invokeServer$0(RestClient.java:68) ~[classes/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]

Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[na:1.8.0_74]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_74]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.8.jar:4.4.8]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar:4.5.3]
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:659) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

If client is not executed in quick succession, no errors.

Debugging / Analysis

Analyzed Wireshark captures. Server is sending RST packets
Not closing the PoolingHttpClientConnectionManager and CloseableHttpClient? Added @PreDestroy shutdown method for graceful shutdown. No luck.
Port exhaustion? Configured for only 200 simultaneous connections.

The Spring RestTemplate configuration

@Configuration
public class Config {

private static final int CONNECT_TIMEOUT = 5000;

private static final int CONNECTION_MANAGER_CONNECTION_REQUEST_TIMEOUT = 0;

private static final int SOCKET_TIMEOUT = 5000;

private static final int MAX_TOTAL = 200;

private static final int MAX_PER_ROUTE = 200;


@Bean
public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() {
    PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
    connectionManager.setMaxTotal(MAX_TOTAL);
    connectionManager.setDefaultMaxPerRoute(MAX_PER_ROUTE);
    connectionManager.setValidateAfterInactivity(CONNECT_TIMEOUT);
    return connectionManager;
}

@Bean
public RequestConfig requestConfig() {
    RequestConfig result = RequestConfig.custom()
            .setConnectionRequestTimeout(CONNECTION_MANAGER_CONNECTION_REQUEST_TIMEOUT)
            .setConnectTimeout(CONNECT_TIMEOUT)
            .setSocketTimeout(SOCKET_TIMEOUT)
            .build();
    return result;
}

@Bean
public CloseableHttpClient httpClient(PoolingHttpClientConnectionManager poolingHttpClientConnectionManager, RequestConfig requestConfig) {
    CloseableHttpClient result = HttpClientBuilder
            .create()
            .setConnectionManager(poolingHttpClientConnectionManager)
            .setDefaultRequestConfig(requestConfig)
            .build();
    return result;
}

@Bean
public RestTemplate restTemplate() {
    HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
    factory.setHttpClient(httpClient(poolingHttpClientConnectionManager(), requestConfig()));

    RestTemplate restTemplate = new RestTemplate();
    restTemplate.setRequestFactory(factory);
    return restTemplate;
}

}

The naive client program:

public void invokeServer(long sleepTime, int concurrentThreads) {

    Site site = new Site();
    site.setMerchantId("MERC");
    site.setSiteId("SITE 100");
    CreateSiteRequest request = new CreateSiteRequest();
    request.setSite(site);
    request.setSleepTime(sleepTime);

    ResponseEntity<Site> response = null;
    if (concurrentThreads > 1) {

        ExecutorService executor = Executors.newFixedThreadPool(concurrentThreads);
        Future<ResponseEntity<Site>>[] futures = new Future[concurrentThreads];
        Callable<ResponseEntity<Site>> callable;

        for (int i = 1; i <= concurrentThreads; i++) {

            callable = () -> restTemplate.postForEntity(SAMPLE_SERVER_URL + "site", request, Site.class);

            futures[i - 1] = executor.submit(callable);
        }

        for (int i = 0; i < futures.length; i++) {
            try {
                response = futures[i].get();
            } catch (Exception e) {
                log.error("Thread " + i + " " + e.getMessage(), e);
            }
        }
    }
}

@PreDestroy
private void shutdown() throws IOException {
    client.close();
    connectionManager.close();
}

Appreciate any input. Thanks.

Any luck? I am having the same issue and can't find an answer anywhere. — Gremash, Feb 22 '18 at 22:47

score 0 · Answer 1 · answered Feb 24 '19 at 14:12

0

we are having similar intermittent issues - we moved out of RestTemplate and using HttpUrlConnection.

answered Feb 24 '19 at 14:12

Rajesh Kamineni

1

1

Please provide a codexample of the issue, so we have some context – Kristoffer Tølbøll Feb 24 '19 at 14:36

Spring RestTemplate SocketException Connection reset on quick consecutive executions

Setup

Problem

Debugging / Analysis

1 Answers1