0

I asked a question What is the ...? using /fcselect handler without ranker ID and got the following docs :

"docs": [
  {"id": "100"},  // ranked first
  {"id": "101"},  // ranked second
  ...
  {"id": "198"},  // ranked second from last (99th)
  {"id": "199"}   // ranked last (100th)
]

Then, I created a ranker using the following ground truth:

What is the ...?,199,5,198,4
...

Then, I asked the *same question* using /fcselect handler with the ranker and got the following docs:

"docs": [
  {"id": "100"},  // ranked first
  {"id": "101"},  // ranked second
  ...
  {"id": "199"},  // ranked 30th
  ...
  {"id": "198"}   // ranked 35th
  ...
]

but I expect like the following order:

"docs": [
  {"id": "199"},  // ranked first
  {"id": "198"},  // ranked second
  {"id": "100"},  // ranked third
  {"id": "101"}   // ranked 4th
  ...
]

Is the ranker properly trained?

takehilo
  • 21
  • 4

1 Answers1

3

See here for an answer to a similar question: https://developer.ibm.com/answers/questions/317822/4-stars-answers.html.

With a learning-to-rank approach there are certainly no guarantees that the ranker will move answers marked as 'correct' in the ground truth to the top of the search result. This is because the ranker is not memorizing correct answers, but rather hoping to capture generalizations in the feature value distributions capturing the overlap between queries and search results. To validate ranker training is behaving as expected, you can measure top 1 result accuracy over a large set of queries (different from the queries used during training) and check for an improvement on average.

That said, it is certainly strange that the top two ranked search results do not change at all in response to ranker training. Some things to consider in experimentation to improve performance:

  • Are the number of rows being passed to the ranker sufficiently high (in your example, the parameter should be set to at least 100 since the default is 10).
  • Are the number of rows included during ranker training (when preparing the ground truth file) the same as the number of rows included during runtime (they should match for optimal performance - and it's a setting you can play with for tuning performance)?
  • Is there a lexical gap between the query and the correct answer documents that's likely to confuse the ranker? Could synonyms / stopword removal / lowercasing / stemming etc be incorporated into your index/query analyzers to improve overlap between the query and correct answer?
  • Are there additional features you could add and pass to the ranker during training and runtime that might be better able to capture the overlap between questions and candidate answers from the search result? See here for more info: https://medium.com/machine-learning-with-ibm-watson/developing-with-ibm-watson-retrieve-and-rank-part-3-custom-features-826fe88a5c63?cm_mc_uid=06908383978514879641730&cm_mc_sid_50200000=1488383112#.gtzsdg4k3
chakravr
  • 126
  • 4