8

I have an annotated corpus in the conll2002 format, namely a tab separated file with a token, pos-tag, and IOB tag followed by entity tag. Example:

John NNP B-PERSON

I want to train a portuguese NER model in NLTK, preferably the MaxEnt model. I do not want to use the "built-in" Stanford NER in NLTK since I was already able to use the stand-alone Stanford NER. I want to use the MaxEnt model to use as comparison to the Stanford NER.

I found NLTK-trainer but I wasn't able to use it.

How can I achieve this?

arop
  • 451
  • 1
  • 5
  • 11

1 Answers1

5

Chapters 6 and 7 of the nltk book explain how to train a "chunker" on an IOB-encoded corpus. The example in chapter 7 does NP chunking, but that's incidental-- your chunker will chunk whatever you train it on. You'll need to decide what features are useful for named entity recognition; chapter 6 covers the basics of choosing features for a classifier. Finally, look at the source for the features used by the nltk's own named entity chunker. They'll probably do a pretty good job in Portuguese too; then you can try adding stemming or other Portuguese-specific features.

alexis
  • 48,685
  • 16
  • 101
  • 161
  • 1
    Thank you, I managed to figure it out eventually, check my [github repository](https://github.com/arop/ner-re-pt/wiki/NLTK) for more info on this. – arop Oct 02 '17 at 15:48
  • Glad to hear that. If my answer solved your problem, please "accept" it by clicking on the check mark. – alexis Mar 21 '18 at 15:40
  • 1
    PS. Took a look at your page. That's pretty abysmal performance you got so far... – alexis Mar 21 '18 at 15:42