How to learn the embeddings in Pytorch and retrieve it later

Question

I am building a recommendation system where I predict the best item for each user given their purchase history of items. I have userIDs and itemIDs and how much itemID was purchased by userID. I have Millions of users and thousands of products. Not all products are purchased(there are some products that no one has bought them yet). Since the users and items are big I don't want to use one-hot vectors. I am using pytorch and I want to create and train the embeddings so that I can make the predictions for each user-item pair. I followed this tutorial https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html. If it's an accurate assumption that the embedding layer is being trained, then do I retrieve the learned weights through model.parameters() method or should I use the embedding.data.weight option?

What makes you think these embeddings are pertained? :) I don't know which part you are referring to exactly, but all embeddings I've seen there on a brief look are trained just with the other layers. — MBT, Nov 02 '18 at 19:52
@blue-phoenox I just couldn't see anything that shows it is being trained with other layers. if it is, then how do I feed a new itemID and userID to the model? How do I get the predictions based on those embeddings? How do I get the embeddings for each user and item after it is learned? — Mina, Nov 02 '18 at 20:13
Getting the embeddings is quite easy you call the embedding with your inputs in a form of a `LongTensor` resp. type `torch.long`: `embeds = self.embeddings(inputs)`. But this isn't a prediction, just an embedding. I'm afraid you have to be more specific on your network structure and what you want to do and what exactly you want to know. I'm not aware of your background, but using embeddings is very simple, you might consider taking some time to fully go through some tutorials and try to understand everything or most of it, to get more familiar with PyTorch and NNs. — MBT, Nov 02 '18 at 20:37
@blue-phoenox thanks for your reply. So once I run the model, how do I retrieve the learned weights for the embedding? Do I use just model.parameters() or should I explicitly ask for the embeddings.weights somehow? — Mina, Nov 05 '18 at 20:19

MBT · Accepted Answer · 2018-11-06T10:21:14.817

model.parameters() returns all the parameters of your model, including the embeddings.

So all these parameters of your model are handed over to the optimizer (line below) and will be trained later when calling optimizer.step() - so yes your embeddings are trained along with all other parameters of the network.
(you can also freeze certain layers by setting i.e. embedding.weight.requires_grad = False, but this is not the case here).

# summing it up:
# this line specifies which parameters are trained with the optimizer
# model.parameters() just returns all parameters
# embedding class weights are also parameters and will thus be trained
optimizer = optim.SGD(model.parameters(), lr=0.001)

You can see that your embedding weights are also of type Parameter by doing so:

import torch
embedding_maxtrix = torch.nn.Embedding(10, 10)
print(type(embedding_maxtrix.weight))

This will output the type of the weights, which is Parameter:

<class 'torch.nn.parameter.Parameter'>

I'm not entirely sure what mean by retrieve. Do you mean getting a single vector, or do you want just the whole matrix to save it, or do something else?

embedding_maxtrix = torch.nn.Embedding(5, 5)
# this will get you a single embedding vector
print('Getting a single vector:\n', embedding_maxtrix(torch.LongTensor([0])))
# of course you can do the same for a seqeunce
print('Getting vectors for a sequence:\n', embedding_maxtrix(torch.LongTensor([1, 2, 3])))
# this will give the the whole embedding matrix
print('Getting weights:\n', embedding_maxtrix.weight.data)

Output:

Getting a single vector:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020]], grad_fn=<EmbeddingBackward>)
Getting vectors for a sequence:
 tensor([[ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502]],
       grad_fn=<EmbeddingBackward>)
Getting weights:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020],
        [ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502],
        [-0.5829, -0.1918, -0.8079,  0.6922, -0.2627]])

I hope this answers your question, you can also take a look at the documentation, there you can find some useful examples as well.

https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding

How to learn the embeddings in Pytorch and retrieve it later

1 Answers1