2

I annotated a corpus using pre-trained syntaxnet model (i.e. using Parse McParseface). I am having a problem understanding the output. There are two metrics reproted in the output. Are those for POS tagging and dependency parsing? If yes, which one is POS tagging performance and which one is for dependency parsing performance?

Here is the output:

INFO:tensorflow:Total processed documents: 21710 INFO:tensorflow:num correct tokens: 454150 INFO:tensorflow:total tokens: 560993 INFO:tensorflow:Seconds elapsed in evaluation: 1184.63, eval metric: 80.95% INFO:tensorflow:Processed 206 documents INFO:tensorflow:Total processed documents: 21710 INFO:tensorflow:num correct tokens: 291851 INFO:tensorflow:total tokens: 504496 INFO:tensorflow:Seconds elapsed in evaluation: 1193.17, eval metric: 57.85%

user2161903
  • 577
  • 1
  • 6
  • 22

1 Answers1

1

If you're using https://github.com/tensorflow/models/blob/master/syntaxnet/syntaxnet/demo.sh then the first metric is POS tag accuracy, the second UAS. They are only meaningful if the conll data you input contains gold POS tags and gold dependencies.

calberti
  • 106
  • 3
  • That makes sense. I wonder why it's showing different number of processed documents and total tokens. I have a gold data sets annotated for POS tag and dependency parse. I was not able to run the parser on multiple files but it runs on a single file. I combined all the annotation, which are in CoNLL format into a single file so that I can pass it easily to the parser.The statistics for total number of documents and tokens should be the same. – user2161903 Jan 26 '17 at 15:36
  • I think the difference in the total token number for POS tagging and dependency parsing is that some tokens in the dependency parsing such as punctuation marks are not considered for evaluation. – user2161903 Jan 27 '17 at 02:34