4

I have a large Flask project spread across multiple modules. I want all of those modules to get access to my MongoDB (PyMongo) database connection. According to best practices, I want exactly one connection pool to persist throughout the application (i.e. throughout all modules).

Infeasible Solutions (from other related StackOverflow answers)

  1. I cannot create a global variable and pass that around to each module I call because that is not thread-safe (especially when my db object is not read-only). Example follows.
  2. I cannot create a new connection for every request because that is inefficient.
  3. I cannot use the Flask.g "global" variable because it is wiped off with every request, making it essentially equivalent to point 2.
  4. I cannot simply put the database connection code into a new module (like config.py) and call it when required from other modules because then I would be creating new connections every time I call it from the different modules.

Problem

  1. How do I create a single and persistent MongoDB connection in a thread- and process- safe manner?
  2. How do I give access to this across modules? In other words, how to make it global across modules?
  3. Is there no best-practice about this? The literature is either outdated or non-existent. Please answer for MongoDB (specifically PyMongo, if possible).

Example

This is a minimum working example of how I would create a global variable and use it across modules (Point 1 in Infeasible Solutions).

main.py

from module1 import helper_function

app = Flask(__name__)
db = new_db_connection()

@app.route('/page1'):
def page1():
    db_return = db.query1()
    db.cache = new_value # this makes this entire code thread-unsafe
    return db_return

@app.route('/page2'):
def page2():
    db_return = db.query2()
    db_return2 = helper_function(db) # doesn't seem right that I have to pass this object around all the time
    return db_return

module1.py

def helper_function(db):
    db_return = db.query3()
    return db_return
agdhruv
  • 575
  • 6
  • 20
  • 1
    Why is the global variable solution thread-unsafe? Databases have their lock mechanism. – AnnieFromTaiwan Oct 06 '20 at 03:13
  • ended up here seeking exactly the same info. One thing I've considered is subclassing Flask, giving it a `get_db` method, and accessing it through the `current_app` proxy – thegreatemu Oct 09 '20 at 21:02

1 Answers1

0

I've been in much the same boat, and the answer I came up with was to pass a parameter to every function. If there's a better way of doing it I would love to know.

Rather than pass the db value, I actually pass a custom context object which has the mongoclient db as well as bunch of other helpful methods (such as recording result ids etc).

class Context:

    def __init__(self, cs: str = None) -> None:
        self.cs = cs

        try:
            self.db = pymongo.MongoClient(self.cs).get_database()
        except Exception as e:
            raise ValueError(f'Failed to connect to server using connection string {self.cs}')
Belly Buster
  • 8,224
  • 2
  • 7
  • 20