Spring data elasticsearch scan and scroll paged result is not working?

Question

I have problems using scan and scroll as the pages from scan return strange result. I am not using them exactly as stated in the documentation. In the method getAllExampleItems() I have a while loop on the "page.hasNext()" method, which always return false. The reason for that is the Pageable from the search query is not set on the result page, and the number of pages therefore is always 1. Another thing that is weird is that I set 100 as the result size and get 500 back! Isn't that a bug or what do I do wrong? This is an example service:

package service;

import config.ElasticSearchConfig;
import items.ExampleItem;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.elasticsearch.core.ElasticsearchTemplate;
import org.springframework.data.elasticsearch.core.query.IndexQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.data.elasticsearch.core.query.SearchQuery;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import static org.elasticsearch.index.query.FilterBuilders.typeFilter;
import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;

@Service
public class ExampleService {

@Autowired
private ElasticsearchTemplate searchTemplate;


public List<ExampleItem> getAllExampleItems() {
    List<ExampleItem> allItems = new ArrayList<>();
    String scrollId = searchTemplate.scan(exampleItemSearchQuery(), 1000, false);
    Page<ExampleItem> page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class);
    if (page != null && page.hasContent()) {
        allItems.addAll(page.getContent());
        while (page != null && page.hasNext()) {
            page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class);
            if (page != null && page.hasContent()) {
                allItems.addAll(page.getContent());
            }
        }
    }
    return allItems;
}


private SearchQuery exampleItemSearchQuery() {
    return new NativeSearchQueryBuilder()
            .withQuery(matchAllQuery())
            .withFilter(typeFilter("exampleitem"))
            .withPageable(new PageRequest(0, 100))
            .build();
}


public void saveAllExampleItems(List<ExampleItem> items) {
         List<IndexQuery> indexQueries = new ArrayList<>();
         for (ExampleItem item : items) {
             IndexQuery qry = new IndexQuery();
             qry.setId(item.getId());
             qry.setObject(item);
             qry.setIndexName(ElasticSearchConfig.ITEMS);
             indexQueries.add(qry);
         }
         searchTemplate.bulkIndex(indexQueries);
         searchTemplate.refresh(ExampleItem.class, true);
     }
}

Could somebody explain to me why this does not work, or what I am doing wrong?

score 1 · Answer 1 · answered Apr 07 '15 at 14:35

1

Working example of scan and scroll with Spring data es can be found at TemplateTest Page

Reason you are getting 500 instead of 100 is because searchHit will be always

size * number_of_primary_shard in elasticsearch.

ref : es site

answered Apr 07 '15 at 14:35

Mohsin Husen

1,008
10
17

Nice to get somebody looking at the problem, but you do not answer the main question. I know about the "working" example, as I wrote. The questiion is why page.hasNext() does not work as expected? The size explanation is probably true, but strange as the Spring data set the size and that is not dependant on how many shards there is. – Pierre Hägnestrand Apr 08 '15 at 23:59

score 1 · Answer 2 · edited Jun 06 '17 at 11:18

1

I faced the same problem. Page must be not null and hasNext() is always true. hasContent() can assert the page. Replace hasNext() with hasContent() and try again....

edited Jun 06 '17 at 11:18

xenteros

15,586
12
56
91

answered Jun 06 '17 at 03:25

Bob.Lau

11
2

score 0 · Answer 3 · answered Jun 15 '15 at 22:07

0

I faced the same issue, then I figured that the difference between my code, and the "working example", is that I used page.hasNext(), just like you did above. Remove that and try again.

answered Jun 15 '15 at 22:07

heyu

185
1
13

Spring data elasticsearch scan and scroll paged result is not working?

3 Answers3

Linked