How could I use the code below to go through a folder of documents and get each documents vector value, and then average the overall value?
documents_list = ['Hello, world','Here are two sentences.']
for doc in documents_list:
doc_nlp = nlp(doc)
print(doc_nlp.vector)
for token in doc_nlp:
print(token.text,token.vector)
It seems like you are wanting to get average vectors on a sentence level, but your example is showing a token level vector representation.
Averaging sentence vectors could be done in the following way:
>>> import numpy as np
>>> np.array([nlp(doc).vector for doc in documents_list]).mean(axis=0)
This would return a single averaged vector for all sentences in documents_list
You could achieve the same on a token level by doing the following:
>>> [np.array([token.vector for token in nlp(doc)]).mean(axis=0) for doc in documents_list]
This will give you a list of averaged word vectors across tokens for each sentence. Basically a list of vectors of length len(documents_list)
As a side note, averaging vectors does not really preserve semantic structure as it implicitly makes the claim that the local context is equivalent to it's broader context. Concatenating might be a better choice in a smaller windowed context.
Make sure to test the results for your domain and task, it could work well for your task depending on your assumptions.
I'm not sure what you mean by a document (I'm not familiar with spacy), but if you want the average, then you can just add each vector to a list, and then after the for loop, do:
avg = sum(vectors_list) / len(vectors_list)
