9

I'm using nltk.word_tokenize for tokenizing some sentences which contain programming languages, frameworks, etc., which get incorrectly tokenized.

For example:

>>> tokenize.word_tokenize("I work with C#.")
['I', 'work', 'with', 'C', '#', '.']

Is there a way to enter a list of "exceptions" like this to the tokenizer? I already have compiled a list of all the things (languages, etc.) that I don't want to split.

Suilan Estévez
  • 91
  • 2
  • possible duplicate of [Modify python nltk.word_tokenize to exclude “#” as delimiter](https://stackoverflow.com/questions/35674103/modify-python-nltk-word-tokenize-to-exclude-as-delimiter) – Ramesh Kumar Aug 10 '17 at 17:24
  • 3
    The difference with that question is that OP is not asking to keep the `#` always, but instead just in `C#` and possibly in hundreds of other particular words such as `F#` and similar technical names. – Alejandro Piad Aug 11 '17 at 08:06

1 Answers1

4

The Multi Word Expression Tokenizer should be what you need.

You add the list of exceptions as tuples and pass it the already tokenized sentences: 

tokenizer = nltk.tokenize.MWETokenizer()
tokenizer.add_mwe(('C', '#'))
tokenizer.add_mwe(('F', '#'))
tokenizer.tokenize(['I', 'work', 'with', 'C', '#', '.'])
['I', 'work', 'with', 'C_#', '.']
tokenizer.tokenize(['I', 'work', 'with', 'F', '#', '.'])
['I', 'work', 'with', 'F_#', '.']
Suzana
  • 4,251
  • 2
  • 28
  • 52