0

I am running these codes

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)

There is a message on the console stating...

You are using config.init_device='cpu', but you can also use config.init_device="meta" with Composer + FSDP for fast initialization.

Where do I make this setting?

zoomraider
  • 117
  • 1
  • 9

1 Answers1

1

Here's an example I found in my notes, I'm figuring out Transformers myself so I can't provide any additional insight other than that this code section seems to work, and as you can see the init method of the config class is apparently where "meta" goes. You can of course open up the Libra source in site-packages if you wanted to drill down and understand what that option does in the code.

Example:

import transformers
from transformers import pipeline
from transformers import AutoTokenizer
import torch 


print("starting")
name = 'mosaicml/mpt-30b-instruct'
tokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-30b')
config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
config.init_device = 'meta' # For fast initialization directly on GPU!
print("loading model")
model = transformers.AutoModelForCausalLM.from_pretrained(name, config=config, torch_dtype=torch.bfloat16, trust_remote_code=True)
print("model loaded")
Michael Mior
  • 28,107
  • 9
  • 89
  • 113
Chris Davis
  • 11
  • 2
  • 1
    Thank you for contributing to the Stack Overflow community. This may be a correct answer, but it’d be really useful to provide additional explanation of your code so developers can understand your reasoning. This is especially useful for new developers who aren’t as familiar with the syntax or struggling to understand the concepts. **Would you kindly [edit] your answer to include additional details for the benefit of the community?** – Jeremy Caney Jul 08 '23 at 00:36