SearchTemplate with sorting and paging variables in Elasticsearch 6.6 and NEST 6.6

Question

All, I am trying to invoke a SearchTemplate defined in ES 6.6. The template has the paging variables( from and size) and an emails that I pass in an array. This also has sorting with a custom script logic. When I run this in kibana, I dont see the paging and sorting doesn't work. I would appreciate any help in getting this to work. Please see details below. There are two indices I search using index aliase.

Mappings for person and guest indices are same( Just for making the example simple)

Index Mapping

PUT _template/person_guest_template
{
  "order": 0,
  "index_patterns": ["person*","guest*"],
  "settings": {
    "index": {
      "analysis": {
        "filter": {
          "autoComplete_filter": {
            "type": "edge_ngram",
            "min_gram": "2",
            "max_gram": "20"
          }
        },
        "analyzer": {
          "autoComplete": {
            "filter": ["lowercase", "asciifolding","autoComplete_filter"],
            "type": "custom",
            "tokenizer": "whitespace"
          },
          "default": {
            "filter": ["lowercase", "asciifolding"],
            "type": "custom",
            "tokenizer": "whitespace"
          }
        }
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "_doc": {
      "dynamic": false,
      "properties": {
        "firstName": {
          "type": "keyword",
          "fields": {
            "search": {
              "type": "text",
              "analyzer": "autoComplete",
              "search_analyzer": "default"
            }
          }
        },
        "lastName": {
          "type": "keyword",
          "fields": {
            "search": {
              "type": "text",
              "analyzer": "autoComplete",
              "search_analyzer": "default"
            }
          }
        },
        "email": {
          "type": "keyword"
        },"email": {
      "type": "keyword"
    }
      }
    }
  }
}

SearchTemplate definition

POST _scripts/guest_person_by_email
{
  "script": {
    "from": "{{from}}{{^from}}0{{/from}}",
    "size": "{{size}}{{^size}}5{{/size}}",
    "sort": [
      {
        "_script": {
          "order": "asc",
          "type": "number",
          "script": "return (doc['type'].value == 'person')? 0 : 1;"
        }
      },
      {
        "firstName": {
          "order": "asc"
        }
      },
      {
        "lastName": {
          "order": "asc"
        }
      }
    ],
    "lang": "mustache",
    "source": """
    {
      "query":{
        "bool":{
          "filter":{
            "terms":{
              "email":
              {{#toJson}}emails{{/toJson}}
            }
          }
        }
      }
    }
"""
  }
}

Search using SearchTemplate

GET guest-person/_search/template
{
  "id":"guest_person_by_email",
  "params": {
    "emails":["rennishj@test.com"]
  }
}

Sample Data

PUT person/_doc/1
{
  "firstName": "Rennish",
  "lastName": "Joseph",
  "email": [
    "rennishj@test.com"
  ],
  "type":"person"
}

Invoking the searchtemplate using NEST 6.6

List<string> emails = new List<string>(){"rennishj@test.com"};
var searchResponse = client.SearchTemplate<object>(st => st
    .Index("guest-person")
    .Id("guest_person_by_email")
    .Params(p => p
        .Add("emails", emails.ToArray())
        .Add("from", 0)     
        .Add("size", 50)
    )
);

Observations

When I remove the from, size and sort logic from searchtemplate, it works
Seems like I am placing the sort and from/size variables at the wrong place?

I found a similar post here https://discuss.elastic.co/t/c-nest-5-search-with-template/104074/2 but seems like the GetSearchTemplate and PutSearchTemplate are discontinued on NEST 6.x

Can this be done using searchtemplates? We use some very complex NEST queries and are moving away from NEST and use searchtemplates.

score 1 · Accepted Answer · answered Feb 11 '19 at 02:26

There are several issues

index template defines "email" field mapping twice
index template sets "dynamic" to false but does not contain a "type" field mapping, so the script sort will fail
the entire search request needs to be defined within "source" for the Put Script API call

NEST can be helpful in building correct search requests and using them as the basis for a search template, besides a whole other bunch of reasons for using a client like round robin requests, automatic fail over and retry, etc.

Here's a complete example

private static void Main()
{
    var defaultIndex = "person";
    var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));

    var settings = new ConnectionSettings(pool)
        .DefaultIndex(defaultIndex)
        .DefaultTypeName("_doc");

    var client = new ElasticClient(settings);

    // WARNING: This deletes the index to make this code repeatable.
    // You probably want to remove this if copying verbatim
    if (client.IndexExists(defaultIndex).Exists)
        client.DeleteIndex(defaultIndex);

    var indexTemplateResponse = client.LowLevel.IndicesPutTemplateForAll<PutIndexTemplateResponse>(
        "person_guest_template",
        @"{
          ""order"": 0,
          ""index_patterns"": [""person*"",""guest*""],
          ""settings"": {
            ""index"": {
              ""analysis"": {
                ""filter"": {
                  ""autoComplete_filter"": {
                    ""type"": ""edge_ngram"",
                    ""min_gram"": ""2"",
                    ""max_gram"": ""20""
                  }
                },
                ""analyzer"": {
                  ""autoComplete"": {
                    ""filter"": [""lowercase"", ""asciifolding"",""autoComplete_filter""],
                    ""type"": ""custom"",
                    ""tokenizer"": ""whitespace""
                  },
                  ""default"": {
                    ""filter"": [""lowercase"", ""asciifolding""],
                    ""type"": ""custom"",
                    ""tokenizer"": ""whitespace""
                  }
                }
              },
              ""number_of_shards"": ""3"",
              ""number_of_replicas"": ""1""
            }
          },
          ""mappings"": {
            ""_doc"": {
              ""dynamic"": false,
              ""properties"": {
                ""firstName"": {
                  ""type"": ""keyword"",
                  ""fields"": {
                    ""search"": {
                      ""type"": ""text"",
                      ""analyzer"": ""autoComplete"",
                      ""search_analyzer"": ""default""
                    }
                  }
                },
                ""lastName"": {
                  ""type"": ""keyword"",
                  ""fields"": {
                    ""search"": {
                      ""type"": ""text"",
                      ""analyzer"": ""autoComplete"",
                      ""search_analyzer"": ""default""
                    }
                  }
                },
                ""email"": {
                  ""type"": ""keyword""
                },
                ""type"": {
                  ""type"": ""keyword""
                }
              }
            }
          }
        }");

    // build a prototype search request     
    var searchRequest = new SearchRequest
    {
        From = 0,
        Size = 0,
        Sort = new List<ISort> 
        {
            new ScriptSort
            {
                Order = Nest.SortOrder.Ascending,
                Type = "number",
                Script = new InlineScript("return (doc['type'].value == 'person')? 0 : 1;")
            },
            new SortField
            {
                Field = "firstName",
                Order = Nest.SortOrder.Ascending
            },
            new SortField
            {
                Field = "lastName",
                Order = Nest.SortOrder.Ascending
            }
        },
        Query = new BoolQuery
        {
            Filter = new QueryContainer[] 
            {
                new TermsQuery
                {
                    Field = "email",
                    Terms = new[] { "emails" }
                }
            }
        }
    };

    var json = client.RequestResponseSerializer.SerializeToString(searchRequest);
    // create template from prototype search request
    var jObject = JsonConvert.DeserializeObject<JObject>(json); 
    jObject["from"] = "{{from}}{{^from}}0{{/from}}";
    jObject["size"] = "{{size}}{{^size}}5{{/size}}";    
    json = jObject.ToString(Newtonsoft.Json.Formatting.None);
    // below is invalid JSON, so can only be constructed with replacement
    json = json.Replace("[\"emails\"]", "{{#toJson}}emails{{/toJson}}");

    // add search template
    var putScriptResponse = client.PutScript("guest_person_by_email", s => s
        .Script(sc => sc
            .Lang(ScriptLang.Mustache)
            .Source(json)
        )
    );

    var person = new Person
    {
        FirstName = "Rennish",
        LastName = "Joseph",
        Email = new[] { "rennishj@test.com" }
    };

    // index document
    var indexResponse = client.Index(person, i => i.Id(1).Refresh(Refresh.WaitFor));

    // search
    var searchResponse = client.SearchTemplate<Person>(s => s
        .Id("guest_person_by_email")
        .Params(p => p
            .Add("emails", person.Email)
            .Add("from", 0)
            .Add("size", 50)
        )
    );
}

public class Person 
{
    public string FirstName {get;set;}
    public string LastName { get; set; }
    public string[] Email {get;set;}
    public string Type {get; set;} = "person";
}

The result of the search template request is

{
  "took" : 47,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "person",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : null,
        "_source" : {
          "firstName" : "Rennish",
          "lastName" : "Joseph",
          "email" : [
            "rennishj@test.com"
          ],
          "type" : "person"
        },
        "sort" : [
          0.0,
          "Rennish",
          "Joseph"
        ]
      }
    ]
  }
}

Thanks Russ Cam for taking the time to go in detail. In our case, we will be creating the search template and indexing the documents outside of NEST and will be only invoking it from NEST. Can we still define the size/from and scripted sort in the template and only invoke that? Our queries have too much logic in there and it was quite a difficult task to compose them with NEST. — dotnetdev_2009, Feb 11 '19 at 03:12
Yes, you can define the template and use the Put Script API from any HTTP client :) — Russ Cam, Feb 11 '19 at 03:20

score 0 · Answer 2 · answered Feb 13 '19 at 17:58

Adding the correct SearchTemplate( moving paging and sorting under "source" as Russ Cam pointed out) just in case someone needs it in the future.

POST _scripts/guest_person_by_email
{
  "script": {    
    "lang": "mustache",
    "source": """
     {
       "from": "{{from}}{{^from}}0{{/from}}",
       "size": "{{size}}{{^size}}5{{/size}}",
      "sort": [
      {
        "_script": {
          "order": "asc",
          "type": "number",
          "script": "return (doc['type'].value == 'person')? 0 : 1;"
        }
      },
      {
        "firstName": {
          "order": "asc"
        }
      },
      {
        "lastName": {
          "order": "asc"
        }
      }
    ],
      "query":{
        "bool":{
          "filter":{
            "terms":{
              "email":
              {{#toJson}}emails{{/toJson}}
            }
          }
        }
      }
    }
"""
  }
}

SearchTemplate with sorting and paging variables in Elasticsearch 6.6 and NEST 6.6

2 Answers2