0

Heyall,

I'm working on my MSc thesis in computer science. More specifically, I am doing research on the effects of tuning the hyperparameters of distributional semantic models when used as features in statistical dependency parsers. I am using word2vec, a non-deterministic neural net-based word embedding software. In order to be able to validate my results, I have to quantify the degree of non-determinism in my models.

I do however think that this question can be asked on a more abstract level -- what test can I use to quantify the degree of non-determinism in a statistical model? Say for instance that I get the following results when performing the same experiment five times:

89.5, 89.7, 89.4, 89.5, 89.9

How can I determine "how non-deterministic" my system is and use this as a basis for accepting or discarding the outcome of future experiments?

Henrik Hillestad Løvold
  • 1,213
  • 4
  • 20
  • 46

1 Answers1

2

If by test you mean a significance or hypothesis test, those tests are useless and you can ignore them.

The appropriate way to quantify uncertainty in language parsing or anything else is to express uncertainty as probability. In the context of language parsing, that means constructing a probability distribution over possible ways to parse a given sentence.

If you need to make decisions, you need to supply additional data which express preferences over outcomes (i.e. utility functions). Probability and utility are combined via the so-called expected utility hypothesis: the best action is the one which maximizes expected utility.

A useful introduction to these concepts, using example from many fields, is "Making Hard Decisions" by Robert Clemen. More specific to your problem, a web search for probabilistic language parsing turns up many hits.

You might get more interest in this question on stats.stackexchange.com. There might already answers to related questions there.

Robert Dodier
  • 16,905
  • 2
  • 31
  • 48