Try to run an NLP model with an Electra instead of a BERT model

Question

I want to run the wl-coref model with an Electra model instead of a Bert model. However, I get an error message with the Electra model and can't find a hint in the Huggingface documentation on how to fix it.

I try different BERT models such like roberta-base, bert-base-german-cased or SpanBERT/spanbert-base-cased. All works. But if I try an Electra model, like google/electra-base-discriminator or german-nlp-group/electra-base-german-uncased then it doesn't work.

The error that is displayed:

out, _ = self.bert(subwords_batches_tensor,  attention_mask=torch.tensor(attention_mask, device=self.config.device))
ValueError: not enough values to unpack (expected 2, got 1)

And this is the method where the error comes from:_bertify in line 349.

score 0 · Answer 1 · answered Jul 03 '22 at 14:22

Just remove the underscore _. ELECTRA does not return a pooling output like BERT or RoBerta:

from transformers import AutoTokenizer, AutoModel

def bla(model_id:str):
  t = AutoTokenizer.from_pretrained(model_id)
  m = AutoModel.from_pretrained(model_id)

  print(m(**t("this is a test", return_tensors="pt")).keys())

bla("google/electra-base-discriminator")
bla("roberta-base")

Output:

odict_keys(['last_hidden_state'])
odict_keys(['last_hidden_state', 'pooler_output'])

Try to run an NLP model with an Electra instead of a BERT model

1 Answers1