ElasticSearch - score boosting using scripting

Question

We have a specific use-case for our ElasticSearch instance: we store documents which contain proper names, dates of birth, addresses, ID numbers, and other related info.

We use a name-matching plugin which overrides the default scoring of ES and assigns a relevancy score between 0 and 1 based on how closely the name matches.

What we need to do is boost that score by a certain amount if other fields match. I have started to read up on ES scripting to achieve this. I need assistance on the script part of the query. Right now, our query looks like this:

{  
   "size":100,
   "query":{  
      "bool":{  
         "should":[  
            {"match":{"Name":"John Smith"}}
            ]
         }
   },
   "rescore":{  
         "window_size":100,
         "query":{  
            "rescore_query":{  
               "function_score":{  
                  "doc_score":{  
                     "fields":{
                       "Name":{"query_value":"John Smith"},
                       "DOB":{
                        "function":{
                            "function_score":{
                                "script_score":{
                                    "script":{
                                        "lang":"painless",
                                        "params":{
                                            "query_value":"01-01-1999"
                                                 },
                               "inline":"if **<HERE'S WHERE I NEED ASSISTANCE>**"
                             }
                           }
                         }
                       }
                     }
                   }
                 }
               }
             },
             "query_weight":0.0,
             "rescore_query_weight":1.0
           }
         }

The Name field will always be required in a query and is the basis for the score, which is returned in the default _score field; for ease of demonstration, we'll just add one additional field, DOB, which if matched, should boost the score by 0.1. I believe I'm looking for something along the lines of if(query_value == doc['DOB'].value add 0.1 to _score), or something along these lines.

So, what would be the correct syntax to be entered into the inline row to achieve this? Or, if the query requires other syntax revision, please advise.

EDIT #1 - it's important to highlight that our DOB field is a text field, not a date field.

Answer 1

Splitting to a separate answer as this solves the problem differently (i.e. - by using script_score as OP proposed instead of trying to rewrite away from scripts).

Assuming the same mapping and data as the previous answer, a scripted version of the query might look like the following:

POST /employee/_search
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "Name": "John"
          }
        },
        {
          "match": {
            "Name": "Will"
          }
        }
      ]
    }
  },
  "rescore": {
    "window_size": 100,
    "query": {
      "rescore_query": {
        "function_score": {
          "query": {
            "bool": {
              "should": [
                {
                  "match": {
                    "Name": "John"
                  }
                },
                {
                  "match": {
                    "Name": "Will"
                  }
                }
              ]
            }
          },
          "functions": [
            {
              "script_score": {
                "script": {
                  "source": "double boost = 0.0; if (params['_source']['State'] == 'FL') { boost += 0.1; } if (params['_source']['DOB'] == '1965-05-24') { boost += 0.3; } return boost;",
                  "lang": "painless"
                }
              }
            }
          ],
          "score_mode": "sum",
          "boost_mode": "sum"
        }
      },
      "query_weight": 0,
      "rescore_query_weight": 1
    }
  }
}

Two notes about the script:

1. The script uses params['_source'][field_name] to access the document, which is the only way to get access to text fields. This is significantly slower as it requires accessing documents directly on disk, though this penalty might not be too bad in the context of a rescore. You could instead use doc[field_name].value if the field was an aggregatable type, such as keyword, date, or something numeric
2. DOB here is compared directly to a string. This is possible because we're using the _source field, and the JSON for the documents has the dates specified as strings. This is somewhat brittle, but likely will do the trick

Answer 2

Assuming static weights per additional field, you can accomplish this without using scripting (though you may need to use script_score for any more complex weighting). To solve your issue of directly adding to a document's original score, your rescoring query will need to be a function score query that:

1. Composes queries for additional fields in a should clause for the function score's main query (i.e. - will only produce scores for documents matching at least one additional field)
2. Uses one function per additional field, with the filter set to select documents with some value for that field, and a weight to specify how much the score should increase (or some other scoring function if desired)

Mapping (as template)

Adding a State and DOB field for sake of example (making sure multiple additional fields contribute to the score correctly)

PUT _template/employee_template
{
  "index_patterns": ["employee"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "Name": {
          "type": "text"
        },
        "State": {
          "type": "keyword"
        },
        "DOB": {
          "type": "date"
        }
      }
    }
  }
}

Sample data

POST /employee/_doc/_bulk
{"index":{}}
{"Name": "John Smith", "State": "NY", "DOB": "1970-01-01"}
{"index":{}}
{"Name": "John C. Reilly", "State": "CA", "DOB": "1965-05-24"}
{"index":{}}
{"Name": "Will Ferrell", "State": "FL", "DOB": "1967-07-16"}

Query

EDIT: Updated the query to include the original query in the new function score in an attempt to compensate for custom scoring plugins.

A few notes about the query below:

• Setting the rescorers score_mode: max is effectively a replace here, since the newly computed function score should only be greater than or equal to the original score
• query_weight and rescore_query_weight are both set to 1 such that they are compared on equal scales during score_mode: max comparison
• In the function_score query:
  • score_mode: sum will add together all the scores from functions
  • boost_mode: sum will add the sum of the functions to the score of the query

POST /employee/_search
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "Name": "John"
          }
        },
        {
          "match": {
            "Name": "Will"
          }
        }
      ]
    }
  },
  "rescore": {
    "window_size": 100,
    "query": {
      "rescore_query": {
        "function_score": {
          "query": {
            "bool": {
              "should": [
                {
                  "match": {
                    "Name": "John"
                  }
                },
                {
                  "match": {
                    "Name": "Will"
                  }
                }
              ],
              "filter": {
                "bool": {
                  "should": [
                    {
                      "term": {
                        "State": "CA"
                      }
                    },
                    {
                      "range": {
                        "DOB": {
                          "lte": "1968-01-01"
                        }
                      }
                    }
                  ]
                }
              }
            }
          },
          "functions": [
            {
              "filter": {
                "term": {
                  "State": "CA"
                }
              },
              "weight": 0.1
            },
            {
              "filter": {
                "range": {
                  "DOB": {
                    "lte": "1968-01-01"
                  }
                }
              },
              "weight": 0.3
            }
          ],
          "score_mode": "sum",
          "boost_mode": "sum"
        }
      },
      "score_mode": "max",
      "query_weight": 1,
      "rescore_query_weight": 1
    }
  }
}