Flair Elmo Embeddings use in Pytorch model

Question

I am trying to generate the elmo embeddings on a PyTorch model, on every batch iteration, like:

for batch in iterator:
        optimizer.zero_grad()
        embeddings = get_elmo_embeddings(batch.dataset.examples)
        predictions = model(embeddings).squeeze(1)
        target = batch.target

where the get elmo embeddings is something like:

from flair.embeddings import StackedEmbeddings
from flair.embeddings import ELMoEmbeddings
from flair.data import Sentence


elmo_embedding = ELMoEmbeddings('original')
stacked_embeddings = StackedEmbeddings(embeddings = [elmo_embedding])

def get_elmo_embeddings(text):
    flat_list = list()
    for x in text:
      sentence = Sentence(x.text)
      stacked_embeddings.embed(sentence)
    
    return elmo(stacked_embeddings)

in order to feed that embeddings to the model, but the problem is that for a single batch it takes hours to create them.. am I doing something wrong? any suggestion of how to make it better?

Also, when I feed it to the model can it go directly to a GRU or it needs some more processing?

Thank all for your time :)

Flair Elmo Embeddings use in Pytorch model

0 Answers0