1

Crazy question maybe: but I want to download the gpt-2 model framework but I want the weights to be initialized randomly. So as if the model still has to be finetuned on the reddit content (including json, vocab, meta & index files etc). Is this possible?

Kind regards!

m.b
  • 45
  • 1
  • 4

1 Answers1

0

Yes, it possible to initalize randomly and train from scratch. For example for T5, you can do:

from transformers import GPT2Model, GPT2Config

config = GPT2Config.from_pretrained() # config as in the paper
model = GPT2Model(config)
Berkay Berabi
  • 1,933
  • 1
  • 10
  • 26