0

With reference to the Web page below, using the Retrieve & Rank service of IBM Bluemix, we are creating a bot that can respond to inquiries.

Question: After learning the ranker once, based on the user's response to the inquiry, how can we construct a mechanism to continuously learn and improve response accuracy?

Assumption: Because there was no API of R&R service to continuously learn from the inquiry response result of the user, tuning the GroundTruth file, I suppose that it is necessary to periodically perform such a process as training the ranker again.

Tuning contents of assumed GT file:

  • If there is a new question, add a set of questions and answers
  • Increase or decrease the relevance score of responses if there is something that could not be answered well by existing question (If bot answered incorrectly, lower the score, if there is a useful answer, raise the score)
ralphearle
  • 1,696
  • 13
  • 18
Shota.N
  • 1
  • Please clarify your question. Add a pointer to the referenced Web page. Add more detail about how you are building your bot and what you have tried so far. – ralphearle Dec 07 '16 at 14:56
  • Thank you for editing. The referenced Web page is here. http://qiita.com/VegaSato/items/6d2d03d6a8b42adcf87e – Shota.N Dec 09 '16 at 02:52
  • If we want to further learn to R & R learned using R & R's API, what should we do?   1.Add new question   2.Add new answer   3.Update relevance score of question and answer There is "creation" in R & R 's API, but there seems to be no "update". – Shota.N Dec 09 '16 at 04:12
  • "I suppose that it is necessary to periodically perform such a process as training the ranker again" -- I think you've nailed it – Leo Dec 09 '16 at 17:15

1 Answers1

1

In order to continuously learn, you will want to do the following:

  • capture new examples i.e. each user input and corresponding result
  • review those examples and create new ranker examples, adjust relevance scores, etc
  • add those new examples to the ranker
  • retrain the ranker using your new and existing examples

NOTE: Be sure to validate that new updates to the ranker data improves the overall system performance. k-fold validation is a great way to measure this.

All in all, learning is a continuous process that should be repeated indefinitely or until system performance is deemed suffcient.

cmd
  • 11,622
  • 7
  • 51
  • 61