BERT: AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

Question

I am trying to freeze some layers of my masked language model using the following code:

for param in model.bert.parameters():
    param.requires_grad = False

However, when I execute the code above, I get this error:

AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

In my code, I have the following imports for my masked language model, but I am unsure what is causing the error above:

from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

So far, I have tried to replace bert with model in my code, but that did not work.

Any help would be good.

Thanks.

score 0 · Answer 1 · answered Aug 19 '21 at 09:15

If you look at the source code of RobertaForMaskedLM code here, you can observe that there is no object with the name bert. Instead, they have an object roberta which is an object of type RobertaModel

Hence, to freeze the Roberta Model and train only the LM head, you should modify your code as:

for param in model.roberta.parameters():
    param.requires_grad = False

BERT: AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

1 Answers1