How to get more than 10 thousand Documents at a time from Elasticsearch Using Jest client

Question

ElasticSearch by-default gives 10 records, but we can set the size parameter and can get the more than 10 records but there is limit, we can set only 10000 as record size if we use Jest client for Elasticsearch, if its more than 10 thousand then throws Exception.

please help me to get more than 10 thousand records at once in elasticsearch using jest client(java)

score 4 · Answer 1 · answered Mar 24 '17 at 14:25

That limit is there for a reason — quoting from the documentation:

The index.max_result_window which defaults to 10,000 is a safeguard, search requests take heap memory and time proportional to from + size.

Depending on your use-case, there are better alternatives:

Real-time / user facing: Use Search After (and avoid deep pagination) with good response times and reasonable heap usage.
Machine / batch processing (often to read all data): Scroll, which creates a search context and keeps it open for the specified amount of time. The result will also be stable as long as the context is open.

+1, and adding to that, `index.max_result_window` can't be changed anymore from `elasticsearch.yml` file since elasticsearch 5.x. Please refer, https://stackoverflow.com/questions/47766765/elasticsearch-index-max-result-window-config-exception/ — Sufiyan Ghori, May 27 '18 at 23:32

score 3 · Answer 2 · answered Mar 23 '17 at 10:18

3

You can update the max-result window of elasticsearch.

curl -XPUT "http://localhost:9200/my_index/_settings" -d '{ "index" : { "max_result_window" : 500000 } }'

But this will only be persisted until Elastic is restarted. You can make a change

index.max_result_window: 1000000

in elasticsearch.yml file for permanent solution.

answered Mar 23 '17 at 10:18

harshil9968

3,254
1
16
26

Hi Harish thanks for you remarks !!!!!!!! BTW does increasing this max_result_window will lead to any memory issue ? i'm using 5.X version of ES – Shubham Pratap Singh Mar 23 '17 at 10:30
not much, only when you query it will take a little bit of more memory. I use it at 1 million limit and works fine – harshil9968 Mar 23 '17 at 10:42
can we do the same in python @harshil9968 – ak3191 Aug 28 '18 at 18:31
@ak3191 yes , make a post request using python. – harshil9968 Aug 28 '18 at 18:41
I have done that...@harshil9968.. there is already a question.. if you can asnwer that... https://stackoverflow.com/questions/52063840/dumping-elastic-data-into-csv-or-at-any-nosql – ak3191 Aug 28 '18 at 18:55
Unfortunately this doesn't work in 7.x and above – Donny V. Jul 08 '21 at 16:21

How to get more than 10 thousand Documents at a time from Elasticsearch Using Jest client

2 Answers2

Linked