0

I am building a LLM powered chatbot. Using ChromaDb for searching relevant documents and then LLM to answer. Any method to get faster responses ?? Currently it takes around 15 seconds to answer. Around 8 seconds for ChromaDb to find relevant document and 7 to 20 seconds for LLM to answer. Any methods, techniques to make it faster ????

i Expected the speed of answering queries to be below 5 seconds

user22403491
  • 7
  • 2

1 Answers1

1

reduce the chunk size during data load.

text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
texts = text_splitter.split_documents(documents)
iVikashJha
  • 159
  • 1
  • 2
  • 14