I'm using the tiiuae/falcon-40b-instruct off HF, and I am trying to incorporate it with LangChain ReAct. I'm using a regular LLMChain with a StringPromptTemplate that's just the standard Thought/Action/Action Input/Observation prompt with some few shot examples incorporated (I've tried with and without the examples, it doesn't seem to work either way). The LLMChain is part of an LLMSingleActionAgent, which is then run by an AgentExecutor.
The problem is, the executor doesn't seem to be capable of using any tools. Most of the time, it is capable of choosing a correct tool, but will just hallucinate the output of the tool. With debug mode on, it is very clear it most of the time doesn't actually enter the tool/start or tool/end chains. Some prior posts about similar issues recommended raising the model's temperature. With the higher temperature, it maybe 5%-10% of the time will enter the tool chains, but the only thing that's returned is Invalid or incomplete response.
I don't have a good idea exactly where something could be going wrong, but here's how my chain and agents are being defined, as a starting point for anyone interested, and a Github repo with the full code here:
# LLM chain consisting of the LLM and a prompt
llm_chain = LLMChain(llm=local_llm, prompt=prompt)
tool_names = [tool.name for tool in tools]
agent = LLMSingleActionAgent(
llm_chain=llm_chain,
output_parser=CustomOutputParser(),
stop=["\nObservation:"],
allowed_tools=tool_names
)
# handle_parsing_errors="Check your output and make sure it conforms!"
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent,
tools=tools,
verbose=True)
I've tried using structured tools and adjusting the prompt. However, the same error occurs with the executor just not using the tools available. I've attached an image of the error here.Model hallucinating and not using tools.
