I'm trying to learn word embeddings using cosine similarity between differing translations. x1 is the embeddings for words in an English sentence. x2 is the embeddings for words in the correct Inuktitut translation. x3 is the embeddings for words in the wrong Inuktitut translation. I'm trying to maximize the cosine similarity between x1 and x2 and minimize it between x1 and x3. There are many triplets of sentences for which I have to do this. My current training function is the following:
def train(self):
for epoch in range(self.epochs):
total_loss = 0
for x1, x2, x3 in zip(self.x1_list, self.x2_list, self.x3_list):
norm1 = tf.norm(x1, axis=1)
norm2 = tf.norm(x2, axis=1)
norm3 = tf.norm(x3, axis=1)
cos1 = tf.matmul(x1, x2, transpose_b=True) / norm2
cos1 = tf.transpose(tf.transpose(cos1) / norm1)
cos2 = tf.matmul(x1, x3, transpose_b=True) / norm3
cos2 = tf.transpose(tf.transpose(cos2) / norm1)
loss = tf.reduce_mean(cos2) - tf.reduce_mean(cos1)
optimizer = tf.train.AdamOptimizer(learning_rate=self.eta).minimize(loss)
self.session.run(tf.global_variables_initializer())
_, loss = self.session.run([optimizer, loss])
total_loss += loss
if epoch % 5 == 0 or epoch == self.epochs-1:
print("Epoch {}: loss = {}".format(epoch, total_loss))
I don't want to recreate all of the tensorflow operations in each loop iteration, but since x1, x2, and x3 are Variables, I can't use placeholders for them. Is there a way to apply these operations to a list of Variable tensors?
