Elasticsearch max document count for each index

Question

I have two separate indices A and B with alias X. Both indices have same document structure. When I search with size = 20 on alias X , I want to set max document size 5 for index B. The search result should contains max 5 documents from index B. If there is no document in index B, search result should contains 20 documents from index A.

Is there any solution to set max document count per index for searchin across multiple index with alias?

score 2 · Accepted Answer · answered Jun 13 '19 at 08:26

You can achieve that using _msearch API.

Below is a sample query as how you can apply this:

POST myalias/_msearch
{"index" : "myindex_news_sports"}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 1}
{"index" : "myindex_news_economics"}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 1}

Basically, _msearch allows you to search multiple requests using the same API. There are two different queries, where you can specify size for two different indexes.

Note that the results would appear in separate groups of JSONs(two results, one for each query).

Sample Response:

{
  "took" : 4,
  "responses" : [
    {                                  <---- Response for Index 1
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 2,
          "relation" : "eq"
        },
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "myindex_news_sports",
            "_type" : "_doc",
            "_id" : "2",
            "_score" : 1.0,
            "_source" : {
              "fooNested" : {
                "sigma" : 9,
                "theta" : 1
              }
            }
          },
          {
            "_index" : "myindex_news_sports",
            "_type" : "_doc",
            "_id" : "4",
            "_score" : 1.0,
            "_source" : {
              "fooNested" : {
                "sigma" : 9,
                "theta" : 1
              }
            }
          }
        ]
      },
      "status" : 200
    },
    {                                  <---- Response for Index 2
      "took" : 2,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 4,
          "relation" : "eq"
        },
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "myindex_news_economics",
            "_type" : "_doc",
            "_id" : "2",
            "_score" : 1.0,
            "_source" : {
              "fooNested" : {
                "sigma" : 9,
                "theta" : 1
              }
            }
          },
          {
            "_index" : "myindex_news_economics",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "fooNested" : {
                "sigma" : 9,
                "theta" : 1
              }
            }
          }
        ]
      },
      "status" : 200
    }
  ]
}

Not sure if this would be ideal for you, I just hope this helps.

msearch provides pagination per index. But I want to do pagination across two index with alias ? Is there any solution which provides global pagination for two responses in msearch @Kamal ? — mstzn, Jun 13 '19 at 08:33
Not possible. You would have to manage this at client side by sending two requests in a transactional way and then merge the results, which I think is more or less similar to _msearch. May I know if you are sorting your results based on score. — Kamal Kunjapur, Jun 13 '19 at 08:41
Yes,sorting is based on scores. It is not good manage pagination on client side. In every search I want to set max document count for index B. — mstzn, Jun 13 '19 at 08:56
If your sorting is based on `_score`, by doing what you are doing, I'm not sure what your use case is, but the results may not be correct. Because your use case desires showing a document with lesser relevancy value of one index above the ones with more relevant from another index. I like to think that is why when it comes to such use-cases, the ES people have designed `_msearch` instead of having it the way you would want in order to prevent such, in my opinion, wrong order of results. But I'm afraid, although your use case may be valid, what you are asking for is not possible in ES. — Kamal Kunjapur, Jun 13 '19 at 09:08
That functionality does not supported by ES. msearch can be a solution for my case. I will accept your answer. Thank you. — mstzn, Jun 13 '19 at 18:06

Elasticsearch max document count for each index

1 Answers1

Sample Response: