Word To Vec with Spacy, words into same categorie

Question

i try do cluster words into the same category. Therfore i wanted to use Spacy Word2Vec. Its already working with easy words like banana apple and car. It shows the nearly same word.

If the words gets more specific like car, battery, accumulator, accu and so on, if the were more technical, Spacy sends Zero vectos. So these words were not included into the bibliothek.

Do you have some input for me?

Furthermore, i have to do it in german.

Thank you very much Jokulema

Have you looked into this post? https://stackoverflow.com/questions/57658888/how-to-specify-word-vector-for-oov-terms-in-spacy — petezurich, Jan 26 '20 at 08:41
And the term you might want to google for is OOV terms (out of vocabulary). — petezurich, Jan 26 '20 at 08:43

score 1 · Answer 1 · answered Jan 26 '20 at 07:51

1

The documentation says that Word2Vec needs a model in order to work with variety of words.

They also give an example of a model which includes ~ 1 milion words and show how to download it:

python -m spacy download en_core_web_lg

Please read the documentation here: https://spacy.io/usage/spacy-101#vectors-similarity

answered Jan 26 '20 at 07:51

V. Sambor

12,361
6
46
65

Yes I downloaded the model (the German one) but the technical words are not included. So it gives back an empty vector. – Jokulema Jan 26 '20 at 08:25

score 0 · Answer 2 · answered Jan 27 '20 at 06:59

If you need word-vectors for words not in the model you're using, you'll have to either:

find & use a different model that contains those words
train your own model from your own training data, that contains many examples of those words' usages in context

Word To Vec with Spacy, words into same categorie

2 Answers2