0

I am trying to fetch all data from elastic index with scroll id.

SearchQuery searchQuery = new NativeSearchQueryBuilder()
                    .withIndices(ELASTIC_INDEX)
                    .withTypes(DOC_TYPE)
                    .withPageable(PageRequest.of(30500, BATCH_SIZE))
                    .withFields("id", "pid")
                    .build();
users = elasticsearchTemplate.startScroll(SCROLL_TIMEOUT, searchQuery, User.class);

But the pagination is not working. It always returns all the documents from page 0.

There are total 30500 documents in the index, so when i set .withPageable(PageRequest.of(30500, BATCH_SIZE)) with batch size 10, there should not be any data returned, but it still returns all data. Also while using scroll for page=304,size=1000 it should return only last 500 documents but it returns 1000 documents.

What i am doing wrong here?

mahfuj asif
  • 1,691
  • 1
  • 11
  • 32

1 Answers1

0

According to javadoc of org.springframework.data.domain.PageRequest#of(int, int) static constructor:

Params: page – zero-based page index, must not be negative. size – the size of the page to be returned, must be greater than 0

Page index is not document index, so you have only documents/BATCH_SIZE = 30500/10 = 3050 indexed pages, and overflowing this param value may cause unexpected behavior

Not A Number
  • 176
  • 4
  • But in my case, they always return all the data. For example page=3040, size=10 it should return the rest 100 documents, rather it returns all the 30500 documents – mahfuj asif Oct 06 '22 at 18:31