Scroll API misses some documents

Question

I am trying to get all the documents from multiple indexes with Scroll Api but it doesn't return all of them. I found a similar question but op was obviously missing first set of documents. Link to the question: Elasticsearch Search Scroll API doesn't retrieve all the documents from an index

Here is my code:

//Code to get indexes

for (String indexName : indexNames) {
   final Scroll scroll = new Scroll(TimeValue.timeValueSeconds(45L));
   SearchRequest searchRequest = new SearchRequest(indexName);
   searchRequest.scroll(scroll);
   SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

   QueryBuilder query = QueryBuilders.boolQuery()
      .filter(QueryBuilders.termQuery(sourceId, 2))
      .filter(QueryBuilders.rangeQuery(date).gte(01-05-2021).lte(31-05-2021));
   searchSourceBuilder.query(query);
   searchSourceBuilder.size(10000);
   searchRequest.source(searchSourceBuilder);

   SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
   String scrollId = searchResponse.getScrollId();
   SearchHit[] searchHits = searchResponse.getHits().getHits();

   List<Model> model = new ArrayList<>();      

   while(searchHits != null && searchHits.length > 0) {
      for (SearchHit document : searchHits){
         //add document to model list created above
         } //end of for loop

   // insert model list to database

   SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
   searchScrollRequest.scroll(scroll);
   searchResponse = client.scroll(searchScrollRequest, RequestOptions.DEFAULT);
   scrollId = searchResponse.getScrollId();
   searchHits = searchResponse.getHits().getHits();

   } //end of while loop

   ClearScrollRequest clear = new ClearScrollRequest();
   clear.addScrollId(scrollId);

} //end of for loop at the top

Total number of documents I should get is 115 millions but I am missing more than 2 millions documents. I repeatedly checked my code but no idea what I am missing.

Your code doesn't seem complete... where does `scroll` come from in the following statement `searchScrollRequest.scroll(scroll);` ? — Val, Jun 25 '21 at 07:59
From line under the first for loop: final Scroll scroll = new Scroll(TimeValue.timeValueSeconds(45L)); — mcsahin, Jun 25 '21 at 08:18
Oops, my bad sorry. You don't see any errors while the scrolling is taking place, right? Something like "No context exception" or similar — Val, Jun 25 '21 at 08:30

Scroll API misses some documents

0 Answers0