0

For example I have an original sentence. The word barking corresponds to the word that is missing.

Original Sentence : The dog is barking.
Incomplete Sentence : The dog is ___________.

For example, using the BERT model, it predicts the word crying instead of the word barking. How will I measure the accuracy of the BERT Model in terms of how syntactically correct and semantically coherent the predicted word is?

(For an instance, there are a lot of incomplete sentences, and the task is to evaluate BERT accuracy based on these incomplete sentences.)Please help.

alyssaeliyah
  • 2,214
  • 6
  • 33
  • 80

1 Answers1

1

For syntax, you can use for instance English Resource Grammar to decide if a sentence is grammatical. It is the biggest manually curated description of English grammar, you can try an online demo. A grammar (given it has a sufficiently large coverage which they usually don't) refuses to parse ungrammatical sentences, unlike statistical/neural parser that happily parses everything (and usually better than grammars).

Estimating semantic plausibility is a very difficult task and given that BERT is probably one of the best current language models, you cannot use another language model as a reference. There are some academic papers that deal with modeling semantic plausibility, you can start, e.g., with this one from NAACL 2018.

Jindřich
  • 10,270
  • 2
  • 23
  • 44