1

The following code is yielding this error for both GPT2 and Xlnet bases. The download of the bases occurs, but the same error prompt is displayed at the end every time. I am using google colab, by the way. ValueError: 50256 is not in list

`

import nlpaug
import nlpaug.augmenter.word as naw
import nlpaug.augmenter.sentence as nas
text = "The quick brown fox jumped over the lazy dog"

aug_cs_gpt2 = nas.ContextualWordEmbsForSentenceAug(model_type = 'gpt2')
temp = aug_cs_gpt2.augment(text)
print(temp)

aug_cs_xlnet = nas.ContextualWordEmbsForSentenceAug(model_type = 'xlnet')
temp = aug_cs_xlnet.augment(text)
print(temp)

Expecting the augmented text to be printed.

    AttributeError                            Traceback (most recent call last)
<ipython-input-41-6810452650ff> in <module>
      9 
     10 aug_cs_xlnet = nas.ContextualWordEmbsForSentenceAug(model_type = 'xlnet')
---> 11 temp = aug_cs_xlnet.augment(text)
     12 print(temp)

2 frames
/usr/local/lib/python3.8/dist-packages/nlpaug/augmenter/sentence/context_word_embs_sentence.py in _custom_insert(self, all_data)
    148                 # Mask token is needed for xlnet. No mask token for gpt2
    149                 if self.model_type in ['xlnet']:
--> 150                     text += ' ' + self.model.MASK_TOKEN
    151 
    152                 texts.append(text)

AttributeError: 'Gpt2' object has no attribute 'MASK_TOKEN'
Isaac Yves
  • 11
  • 2

1 Answers1

0

This is a known bug in nlpaug in version prior to 0.0.16. You should upgrade nlpaug to a version newer than this one and the issue should be gone.

sophros
  • 14,672
  • 11
  • 46
  • 75