Open distro for elasticsearch

Question

can we fetch more than 10000 records from open distro elasticsearch through java ?

I tried with normal basic sql query :

select id, name from employee order by id desc limit 10, 10;

if I try to fetch data with the above query it works fine but it's range should be under 10k. If I change limit (from, size) ie limit 10000, 10 then it does not return any value.

Answer 1

Yes, you can, the Default value of index.max_result_window which controls this limit is 10K, hence you are not getting the result, please read the desc of this setting in official ES docs

The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this.

When you give from and size as 10k,10, it crosses the limit of from+size of 10k. but the good news is that its a dynamic setting and can be changed without restarting the ES nodes:

PUT http://{{esHost}}:{{port}}/{{index}}/_settings

{
    "index.max_result_window" : 20000
}

Check if it's updated in the index settings

GET http://{{eshost}}:{{port}}/{{index}}/_settings

{
    "64142048": {
        "settings": {
            "index": {
                "number_of_shards": "1",
                "provided_name": "64142048",
                "max_result_window": "20000", --> note the change
                "creation_date": "1601531937379",
                "number_of_replicas": "1",
                "uuid": "TyEBaknxRJy9NGgma6jOmA",
                "version": {
                    "created": "7080099"
                }
            }
        }
    }
}