In this post (https://stackoverflow.com/a/64526124/15693663) I see a very simple and short solution to invert the embedding layer. I used the inverse embedding layer, but it does not update the weights in the network. The proposed inverse embedding layer is copied from the post here (bellow):
import torch
embeddings = torch.nn.Embedding(1000, 100)
my_sample = torch.randn(1, 100)
distance = torch.norm(embeddings.weight.data - my_sample, dim=1)
nearest = torch.argmin(distance)
What i did: I used an embedding layer that gets one input and generated 16D output. Then, I add two hidden dense layers (64->16) and one inverse embedding layer. In short
X -> embedding ->dense layer(64D)->dense layer(16D) -> inverse embedding -> X'
X and X' are integer numbers.
To compute the loss, I used torch.norm(X - X'). But it does not update the weights. I can not figure out the problem and why there is no update in weights.
A short implementation is shown bellow:
# lS_o = Offset, lS_i = input number
optimizer = opts['sgd'](parameters, lr=args.learning_rate)
#--------------------------------------
model forward(self, lS_o, lS_i):
out_emb1 = self.embl_inp(lS_o, lS_i) # 16D == embedding layer
out_dl1= self.DLyr1(out_emb1) # 64D == Dense Layer 1
out_dl2 = self.DLyr2(out_dl1) # 16D == Dense Layer 2
ly = out_dl2
distance = self.emb_out.weight.data-ly[i,None] #subtract each row of weight matrix with each row in ly
out = torch.argmin(torch.norm(distance, dim=1), dim=0)
return torch.stack(out)
train_ds = Dataset(...
train_ld = DataLoader(train_ds, ...
pbar = tq.tqdm(enumerate(train_ld, total=len(train_ld))
for j, inputBatch in pbar:
lS_o, lS_i = unpack_batch(inputBatch)
ae_out = model(lS_o, lS_i, use_gpu=True)
loss = torch.norm(ae_out - lS_i)
optimizer.zero_grad()
loss.backward()
optimizer.step()
I have studied the gumbel_softmax because argmin and argmax are not differentiable. So I have changed the code in the "model forward()" function as the following:
distance = torch.norm(distance, dim=1)
out = 1 - torch.nn.functional.gumbel_softmax(dist)
Because I am going to find the minimum, I subtract the output of the gumbel_softmax by 1.
