How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

Question

Am using a Spring Boot 1.5.4.RELEASE Microservice to connect to an ElasticSearch 5.5.0 instance using the low level Rest Client that ElasticSearch provides.

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>

<dependencies>
    <!-- Spring -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>

    <!-- Apache Commons -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>

    <!-- Jackson -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>2.8.9</version>
    </dependency>

    <!-- Log4j -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>

    <!-- JUnit -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>

    <!-- Swagger -->
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger2</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger-ui</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

Everything is setup correctly but after a bunch of hits, client apps were reporting an HTTP 500 error and this is what appeared in the log files:

java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[na:1.8.0_141]
        at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.8.0_141]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.<init>(AbstractMultiworkerIOReactor.java:142) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.<init>(DefaultConnectingIOReactor.java:79) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.client.IOReactorUtils.create(IOReactorUtils.java:43) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:666) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:202) ~[rest-5.5.0.jar!/:5.5.0]
        at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:180) ~[rest-5.5.0.jar!/:5.5.0]
        at com.myapp.controller.SearchController.getSearchQueryResults(SearchController.java:94) ~[classes!/:1.0]

Inside SearchController (the second line after the // comment is line 94):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup RestClient
        RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();

        // Setup query and send and return ResponseEntity...

    }
}

Its obvious that what it was never closed after calling the restClient.performRequest() method...

So, I put this into my code:

Response response = null;
try {
   // Submit Query and Obtain Response
   response = restClient.performRequest("POST", endPoint,  Collections.singletonMap("pretty", "true"), entity);
}
catch (IOException e) {
   LOG.error("\n\n\tException: " + e + "\n\n");
   e.printStackTrace();
}
finally {
   restClient.close();
}

Read on Elastic Search's documentation that the RestClient class is thread-safe...

Also, read about the restClient.performRequestAsync() method but am somewhat inexperienced with threads and the description inside the documentation is vague.

Question(s):

Is my solution the best way to handle and close a bunch of socket resources?
Would appreciate if someone could show me a better way to use the low level RestClient with Elastic Search in sense that it won't cause the same issue with the socket resources not being freed resulting in an HTTP 500. Should I be using restClient.performRequestAsync? Could someone please provide an example?

Thank you for taking the time to read this...

I think restClient.close() in finally block should do the trick. Isn't it working? — Hatim Stovewala, Sep 15 '17 at 08:24
Can you confirm that you're not using the spring boot starters for Elasticsearch? — Val, Sep 15 '17 at 08:45
The rest client should be created only once in a configuration bean, injected as a dependency into your controllers, and that's it. You don't need to create a new instance for each request. — Val, Sep 15 '17 at 08:46
@Val, I updated the post to give you the full pom.xml file dependencies which should answer your question regarding Spring Boot Starters. I am using Elastic Search's Raw Rest Client. Can you show me an example of the configuration bean via code? By the way, I am not creating a new instance (its the Builder pattern) I am not using the "new" Java keyword. I would really appreciate if you could provide code. There might be 1000s of mobile apps (both iOS & Android) which will be creating a single connection to this Spring Boot Microservice. So performance is critical. As is non-blocking. — PacificNW_Lover, Sep 15 '17 at 09:18
Where and when do you create your restClient. You say "inside `SearchController`", but it's not clear when. — Val, Sep 15 '17 at 09:21
@HatimStovewala, It hasn't failed yet with the restClient.close() in a the finally block, thanks. Like I told Val, that since the target clients are native apps that run on iOS and Android (and it might be thousands of connections) to this Spring Boot Microservice should I be using threads or restClient.performRequestAsync()? Sorry - need the best way to accommodate these numerous mobile clients. — PacificNW_Lover, Sep 15 '17 at 09:21
@Val, I just edited my post and displayed how I am using the raw Rest Client inside my SpringBoot Microservice. How can I setup it up to accommodate 1000s of single connections from mobile devices? Is this try / catch / finally good enough or do I need to use threads or the restClient.performRequestAsync() method call? If you can provide me code regarding the Configuration Bean along with dependency injection, I would be very grateful. — PacificNW_Lover, Sep 15 '17 at 09:31

score 12 · Accepted Answer · answered Sep 15 '17 at 09:46

It's is not a good practice to create a RestClient on every single request. You should create a single instance via a configuration bean like the one below:

@Configuration
public class ElasticsearchConfig {

    @Value("${elasticsearch.host}")
    private String host;

    @Value("${elasticsearch.port}")
    private int port;

    @Bean
    public RestClient restClient() {
        return RestClient.builder(new HttpHost(host, port))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();
    }
}

And then in your SearchController class you can inject it like this (and also add a cleanup method to close the restClient instance when your container goes down):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestClient restClient;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup query and send and return ResponseEntity...

        Response response = this.restClient.performRequest(...);

    }

    @PreDestroy
    public void cleanup() {
        try {
            logger.info("Closing the ES REST client");
            this.restClient.close();
        } catch (IOException ioe) {
            logger.error("Problem occurred when closing the ES REST client", ioe);
        }
    }

}

Thank you for such a well designed approach! Would I still have to use restClient.close(); inside a finally { } or does the cleanup() method do that? The reason I got errors before was because I wasn't invoking restClient.close(). Should I include the finally after performRequest() and also keep the cleanup() method? — PacificNW_Lover, Sep 16 '17 at 00:22
No, you can keep the rest client open all the time and only close it in the cleanup method. Try it out an you'll see ;-) — Val, Sep 16 '17 at 03:20
It happend again, this time using your solution! It seemed like the RestClient never closed its connection to begin with. I created a new post with my findings here: https://stackoverflow.com/questions/46559512/elasticsearch-rest-client-still-giving-ioexception-too-many-open-files — PacificNW_Lover, Oct 04 '17 at 07:42

How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

1 Answers1

Linked