Loading Google Flan models through Langchain HuggingFacePipeline throws error of unrecognized configuration class for AutoModelForCausalLM

Question

Trying to load any Google Flan model through Langchain HuggingFacePipeline as shown below

from langchain import HuggingFacePipeline
llm = HuggingFacePipeline.from_model_id(model_id="google/flan-t5-xl", task="text-generation", model_kwargs={"temperature":0, "max_length":64})

throws a ValueError as

ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM. Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

It flows through the following lines of code.

It works fine if I load it through Langchain HuggingFaceHub though.

Any tips on fixing it while still using HuggingFacePipeline module is appreciated.

score 1 · Answer 1 · answered May 08 '23 at 22:38

1

Change the task parameter from text-generation to text2text-generation.

(google/flan-ul2 is based on an AutoModelForSeq2SeqLM instead of an AutoModelForCausalLM used by the Bloom model in the current Langchain doc example.)

answered May 08 '23 at 22:38

G__

7,003
5
36
54

score 0 · Answer 2 · answered Aug 26 '23 at 14:11

Code for setting up HuggingFace pipeline

    from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, AutoConfig
    from langchain.llms import HuggingFacePipeline
    from transformers import pipeline



    model_id = 'google/flan-t5-small'

    config = AutoConfig.from_pretrained(model_id)
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForSeq2SeqLM.from_pretrained(model_id, config=config)

    _pipeline = pipeline('text2text-generation',
                    model=model,
                    tokenizer=tokenizer,
                    max_length = 512
                    )

    hf_llm = HuggingFacePipeline(pipeline = _pipeline)

Loading Google Flan models through Langchain HuggingFacePipeline throws error of unrecognized configuration class for AutoModelForCausalLM

2 Answers2