I have this Opensearch Vector DB and I maintain multiple indices that start with "index-" (for example index-pdf, index-html). I have indexed sets of documents to each of the indices using Langchain's OpenSearchVectorSearch.from_documents() function.
Now, I want to run some queries which I want them to be run across multiple indices. An example would be "What is the title of each document?". When I execute the below code, it either just outputs answer from the first or last matching index, or says it cannot find the answer. Here is my current code:
from langchain.vectorstores import OpenSearchVectorSearch
from langchain.chains import RetrievalQA, ConversationalRetrievalChain
import os
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
embeddings = OpenAIEmbeddings()
def get_llm(model):
llm = ChatOpenAI(model_name=model.lower(), verbose=False, temperature=0)
return llm
docsearch = OpenSearchVectorSearch(opensearch_url="http://localhost:9200",
index_name="index-*",
embedding_function=embeddings)
chain = ConversationalRetrievalChain.from_llm(
llm=get_llm("gpt-3.5-turbo"),
retriever=docsearch.as_retriever(),
)
result = chain({'question': 'What is the title of each document?', "chat_history": []})
response = result['answer']
print(response)
How can I effectively run a Langchain based query on multiple indices (or all indices) and get a response back? Sample questions could be listing all document names/titles.
