Stanford NER Tagger super slow

Asked Nov 08 '16 at 19:35

Active Aug 29 '17 at 07:54

Viewed 677 times

I am trying to process a text file of 1M lines. Each string is one line, and I need to obtain location information from each string. The easiest way I thought of doing so was using Stanford's NER tagger in PYTHON.

However, the tagger is performing extremely slow, like 2 seconds per line. By this speed it will take 20 days to complete my file. Also its somewhat inaccurate, missing obvious locations like "nyc".

Anyone know how I can speed things up a little and make it more accurate?

Thanks

asked Nov 08 '16 at 19:35

Stanford Core NLP is Java. How are you calling it from python? – bmargulies Nov 08 '16 at 19:43
There is an interface that allows python to access it from nltk.tag import StanfordNERTagger – Nov 08 '16 at 19:44
How fast is it in Java? – Nov 08 '16 at 19:44
Not very fast, but who knows? You need to test it on your own data. – bmargulies Nov 08 '16 at 19:45
Same speed as Python? Like 2 seconds per line...? I would test it, but I have to setup all the java stuff. Troublesome – Nov 08 '16 at 19:45
http://stackoverflow.com/a/23322996/610569 – alvas Nov 09 '16 at 01:59
Why are you testing tagging speed per sentence? You should be testing it per corpus with `tag_sents()`. – alvas Nov 09 '16 at 02:00
Im not testing tagging speed per sentence. My file is 1million lines, I need to extract LOCATION information from each line. And so far its not working. – Nov 09 '16 at 14:35

Stanford NER Tagger super slow

0 Answers0