Is there an algorithm or technique in place that is capable of clustering words based on meaning?

Question

I need to develop clusters for similar words based on the meaning of the word. For example, I want "apple" in the same cluster as "fruit," "banana," "honeycrisp."

Is there some lexicon package that has something like this in place, or would it be up to me to create my own clusters?

score 0 · Answer 1 · answered May 01 '19 at 22:42

0

Google's natural language processing api might help. Here is a link: https://cloud.google.com/natural-language/. There is option to demo it right on the site, so you can see if that is what you are looking for.

answered May 01 '19 at 22:42

marsnebulasoup

2,530
2
16
37

score 0 · Answer 2 · answered May 01 '19 at 23:48

There are a number of pretrained models for you to download that are vector representations of word stems. A popular choice is Google's pretrained 300-dimensional Word2Vec Model which can be downloaded from:

https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit

and loaded (after unzipping) with:

import gensim
model = gensim.models.Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)

This model is quite large, but does exactly what you want. If you are interested in only a subset of words, I suggest that you extract only those from the model and store their coordinates in a DataFrame for later (and faster) use.

For other models see the following:

Is there an algorithm or technique in place that is capable of clustering words based on meaning?

2 Answers2