Java High Level Rest Client Memory Usage In Range Queries

Question

Following function run range queries on the elastic search, when I use this function repeatedly in a multithread environment, The memory increases drastically until the application crashes.

Does anyone have a solution to this problem?

/**
  RestHighLevelClient restHighLevelClient = new RestHighLevelClient( //
                RestClient.builder(new HttpHost("localhost",9200,"http")));
**/
public List<Map<String, Object>> getAllDocumentsInRange(long startTime, long endTime,
       RestHighLevelClient restHighLevelClient) {
    try {
        QueryBuilder queryBuilder = QueryBuilders//
                .rangeQuery("date")//
                .gte(startTime)//
                .lte(endTime);


        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        searchSourceBuilder.query(queryBuilder);

        searchSourceBuilder.sort(new FieldSortBuilder("date").order(SortOrder.ASC));

        searchSourceBuilder.size(10000);

        searchSourceBuilder.timeout(new TimeValue(20, TimeUnit.SECONDS));

        List<Map<String, Object>> docsMap = new ArrayList<>();

        SearchRequest searchRequest = new SearchRequest(new String[] { "mainIndex" }, searchSourceBuilder);

        Scroll scroll = new Scroll(TimeValue.timeValueSeconds(30));
        searchRequest.scroll(scroll);


        RestHighLevelClient restHighLevelClient = new RestHighLevelClient( //
                RestClient.builder(new HttpHost("localhost",9200,"http")));

        SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);

        String scrollId = searchResponse.getScrollId();

        SearchHit[] searchHits = searchResponse.getHits().getHits();

        while (searchHits != null && searchHits.length > 0) {

            for (SearchHit hit : searchHits) {
                Map<String, Object> elasticDocVersion = hit.getSourceAsMap();
                docsMap.add(elasticDocVersion);
            }

            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(scroll);

            searchResponse = restHighLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);

            scrollId = searchResponse.getScrollId();

            searchHits = searchResponse.getHits().getHits();
        }

        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = //
                restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
        boolean succeeded = clearScrollResponse.isSucceeded();
        logger.trace("search scroll clearation:{}", succeeded);
    } catch (Exception e) {
        logger.error("error in creaing QueryBuilder class: {}", e.getMessage());
    }
    return new ArrayList<>();
}

this is my memory usage image after running the application

I have tried different solutions, like synchronize the above function but none of them solved the problem!

score 1 · Answer 1 · answered Oct 08 '20 at 18:00

Not sure what elasticsearch client version you are using, but you may be interested in this elasticsearch rest client memory leak.

Also, I'm not sure which one is intended the restHighLevelClient that is passed in or the one that is created in the method. Only one is valid, but in any case, I would suggest calling restHighLevelClient.close() or using it in a try-with-resources when done with the client.

try (RestHighLevelClient restHighLevelClient = 
   new RestHighLevelClient(RestClient.builder(new HttpHost("localhost",9200,"http"))) {
...
}

Java High Level Rest Client Memory Usage In Range Queries

1 Answers1