3

My goal is to create text generator which is going to generate non-english text based on learning set I provide to it.

I'm currently at the stage of figuring out how the model actually should looks like. I'm trying to implement fasttext pre-trained model as an Embedding layer in my net. But due to that I have some questions.

1) How to properly prepare fasttext model? Should I just download vectors, for the language that I need, and include them in the project, or I have to build it first using skipgram or cbow or in some other way?

2) How am I suppose to exchange Keras Embedding() with fasttext model?

Now I have something like this: 

    model = Sequential()
    model.add(Embedding(vocabulary_size, seq_len, input_length=seq_len, output_dim=OUTPUT_DIM))
    model.add(LSTM(50, return_sequences=True))

And instead of model.add(Embedding()) I wish to put fasttext vector.

I hope I explained it clearly.

pawcio
  • 71
  • 1
  • 5

1 Answers1

3

If you do not plan to finetune the embedding, I would just load the FastText embeddings, turn each sentence into a 2-D (length × embedding dim.) tensor, and use those as an input to the network.

If you want to fine-tune the FastText embeddings, they, of course, need to be part of model in Keras. In that case, you need to initialize the Embedding layer with an embedding matrix extracted from your FastText model (you just stack all the ord vectors). This is done using a constant initializer. In any case, you also need to keep a dictionary that will convert the words into indices in the embedding matrix.

Note that the second argument of Embedding constructor is not the sequence length (as in your code snippet), but the number of words in your vocabulary.

Jindřich
  • 10,270
  • 2
  • 23
  • 44