1

How can I fine tune pre-trained embeddings in embedding layer in tf.keras?

# embedding layer with pre trained weights
embedding_layer = layers.Embedding(
    input_dim=self.vocab_size + 2,
    output_dim=self.emb_size,
    embeddings_initializer=initializers.Constant(embedding_matrix),
    mask_zero=mask_zero,
    trainable=False
)

If I just change trainable = True will it fine tune the pre-trained embeddings that I have? Or should I also have to remove initializers.Constant as initializer?

Christian Baumann
  • 3,188
  • 3
  • 20
  • 37
user_12
  • 1,778
  • 7
  • 31
  • 72

1 Answers1

2

You can refer to the following answer:

When setting the trainable to true you let the embedding layer to fine-tune.

Setting the embeddings_initializer will contradict the trained flag. you should either don't set it to constant, or you can just set the weights with the embedding_matrix. You can refer to the following links:

  1. setting the weights
  2. meaning of constat

Regarding to your comment where is the weights refer to the following link: Keras Embedding ,where is the "weights" argument?

David
  • 8,113
  • 2
  • 17
  • 36
  • In the official docs, I couldn't find a weights parameter io embedding layer? Am I missing something here – user_12 Oct 03 '20 at 12:33
  • Also I am bit confused, are word embeddings called as weights? – user_12 Oct 03 '20 at 12:35
  • It’s inherit it from the base class, it’s “hidden” in the **kwargs. You could also can’t find name but it exist – David Oct 03 '20 at 12:37
  • Just one more doubt, are work embeddings is called as weights? Because we are literally passing word embedding matrix to weights parameter right? – user_12 Oct 03 '20 at 12:41
  • @user_12 I don’t understand the question, should you mind to elaborate. Maybe the following will help https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html – David Oct 03 '20 at 12:42
  • Are word embeddings matrix and weights are same? Because you mentioned to use weights parameter in embedding layer and pass embeddings matrix as value. So, I'm bit confused of the terminology. Are weights and embeddings terminology interchangeable used. – user_12 Oct 03 '20 at 12:44
  • @user_12 Once you have an embedding layer, trained, this matrix can serve as weights. It will be a matrix of numbers, so you can always use them as the starting weights if you wish to – David Oct 03 '20 at 12:45
  • I think I got it, so you are saying we can use those embeddings as starting weights and fine-tune it. – user_12 Oct 03 '20 at 12:57