Chroma document retrieval in langchain not working in Flask frontend

Question

I am using langchain to create a chroma database to store pdf files through a Flask frontend. I am able to query the database and successfully retrieve data when the python file is ran from the command line.

I am however trying to use the the feature through Flask, and so far I can't get it to work. This is the class I am using to query the database:

from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
from langchain.vectorstores import Chroma

class Chat_db:
    def __init__(self):
        persist_directory = 'chromadb'
        embedding = OpenAIEmbeddings()
        vectordb = Chroma(persist_directory=persist_directory, embedding_function=embedding)
        retriever = vectordb.as_retriever(search_kwargs={"k": 2})
        self.qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff",
                                         retriever= retriever)
        
    def chat_over_documents(self, query):
        result = self.qa({"query": query})
        return result['result'] 

#if __name__ == "__main__":
#    query = "Where was the US declaration of independence signed?"
#    vector_db = Chat_db()
#    result= vector_db.chat_over_documents(query)
#    print(result)

If I uncomment the "main" section, I get the result I expect. If I run the query through 'app.py', I get the canned langchain result 'I'm sorry, don't know'. This is how I am calling it from 'app.py'

## app.py
...
chat_db = Chat_db()
@app.route('/message', methods=['POST'])
def message():
    user_message = request.json['message']
    if "document" in user_message.lower():
        response = chat_db.chat_over_documents(user_message)
        return jsonify({"message": response})
    else:
        response = assistant.chat_with_gpt(user_message)
        return jsonify({"message": response})


...
#DEBUG LOGS
DEBUG:openai:message='Request to OpenAI API' method=post path=https://api.openai.com/v1/embeddings
DEBUG:openai:api_version=None data='{"input": [[6190, 3371, 757, 264, 308, 7420, 1122, 79538]], "model": "text-embedding-ada-002", "encoding_format": "base64"}' message='Post details'
DEBUG:urllib3.util.retry:Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.openai.com:443
DEBUG:urllib3.connectionpool:https://api.openai.com:443 "POST /v1/embeddings HTTP/1.1" 200 None
DEBUG:openai:message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=63 request_id=... response_code=200
DEBUG:chromadb.config:Starting component PersistentLocalHnswSegment
DEBUG:openai:message='Request to OpenAI API' method=post path=https://api.openai.com/v1/completions
...

I am not sure what I am doing incorrectly; any help is highly appreciated.

I have tried tracing data using the browser's developer tool and debugging in terminal. From the logs, my suspicion is that langchain is making two openai api calls even before Chroma db completes initializing.

Answer 1

If you db initialization is taking time then, you can check flag to make sure initialization is complete

        class Chat_db:
            def __init__(self):
                self.initialized = False  # Initialization flag
                self.initialize()

            def initialize(self):
                try:
                    persist_directory = 'chromadb'
                    embedding = OpenAIEmbeddings()
                    vectordb = Chroma(persist_directory=persist_directory, embedding_function=embedding)
                    retriever = vectordb.as_retriever(search_kwargs={"k": 2})
                    self.qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=retriever)
                    self.initialized = True  
                except Exception as e:
                    print(f"An error occurred during initialization: {e}")

        @app.route('/message', methods=['POST'])
        def message():
             if not chat_db.initialized:
                 return jsonify({"message": "The system is still initializing. Please try again later."}), 503