1

The BERT-based SRL model that Shi and Lin develop (which is currently the the backend of the AllenNLP SRL model) has more consistent advantages over Ouichi et al.'s (2018) ensemble model when using BERT-large, instead of BERT-base. For example, the Shi and Lin model gets better F1 than Ouichi on CoNLL 05 only when using BERT-Large.

So, is it possible to use the AllenNLP SRL model with BERT-large rather than BERT-base?

Russell Richie
  • 421
  • 1
  • 5
  • 15
  • 1
    This is definitely possible, you'd just need to fine-tune it using "bert-large". You could easily adapt this training config to do so: https://github.com/allenai/allennlp-models/blob/main/training_config/structured_prediction/bert_base_srl.jsonnet – petew Aug 06 '21 at 22:22
  • Thanks, I suspected it might be something like that. Is it just a matter of changing that first line to `local bert_model = "bert-large-uncased";`? – Russell Richie Aug 07 '21 at 14:33
  • 1
    Yeup! Although you may need to adjust some hyperparameters like the learning rate. – petew Aug 09 '21 at 15:46
  • 1
    Thanks! It seems to work (or at least, I get other errors seemingly unrelated to the base vs large, lol). As for the learning rate, the Shi and Lin paper seems to imply the same learning rate for base and large.... – Russell Richie Aug 09 '21 at 16:00
  • @petew okay you seem like an informed person about this ;-) so here's another q: I tried fine-tuning adamlin/ClinicalBert_all_notes and got an error like `ValueError: Unrecognized model in adamlin/ClinicalBert_all_notes. Should have a 'model_type' key in its config.json`. I guess that means this file, `https://huggingface.co/adamlin/ClinicalBert_all_notes/blob/main/config.json`, but I can't edit it on huggingface.com. Does this mean I should download the model files, edit them locally, and load locally? How would I point to local model in the training config jsonnet? – Russell Richie Aug 20 '21 at 18:56
  • 1
    Yes you should be able to download those files locally. Then in your AllenNLP config, change the name of the pretrained model to the local path of your downloaded files (the directory). – petew Aug 23 '21 at 15:46
  • 1
    Glad to hear it! – petew Aug 24 '21 at 22:54

0 Answers0