Call method once when Flask app started despite many Gunicorn workers

Question

I have a simple Flask app that starts with Gunicorn which has 4 workers.

I want to clear and warmup cache when server restarted. But when I do this inside create_app() method it is executing 4 times.

def create_app(test_config=None):
    app = Flask(__name__)

    # ...  different configuration here

    t = threading.Thread(target=reset_cache, args=(app,))
    t.start()
    return app

[2022-10-28 09:33:33 +0000] [7] [INFO] Booting worker with pid: 7
[2022-10-28 09:33:33 +0000] [8] [INFO] Booting worker with pid: 8
[2022-10-28 09:33:33 +0000] [9] [INFO] Booting worker with pid: 9
[2022-10-28 09:33:33 +0000] [10] [INFO] Booting worker with pid: 10
2022-10-28 09:33:36,908 INFO     webapp reset_cache:38      Clearing cache
2022-10-28 09:33:36,908 INFO     webapp reset_cache:38      Clearing cache
2022-10-28 09:33:36,908 INFO     webapp reset_cache:38      Clearing cache
2022-10-28 09:33:36,909 INFO     webapp reset_cache:38      Clearing cache

How to make it only one-time without using any queues, rq-workers or celery? Signals, mutex, some special check of worker id (but it is always dynamic)? Tried Haven't found any solution so far.

score 0 · Answer 1 · answered Nov 03 '22 at 16:09

I used Redis locks for that.

Here is an example using flask-caching, which I had in project, but you can replace set client from whatever place you have redis client:

import time

from webapp.models import cache  # cache = flask_caching.Cache()


def reset_cache(app):
    with app.app_context():
        client = app.extensions["cache"][cache]._write_client  # redis client

        lock = client.lock("warmup-cache-key")
        locked = lock.acquire(blocking=False, blocking_timeout=1)
        if locked:
            app.logger.info("Clearing cache")
            cache.clear()

            app.logger.info("Warming up cache")
            # function call here with `cache.set(...)`
            app.logger.info("Completed warmup cache")

            # time.sleep(5)  # add some delay if procedure is really fast
            lock.release()

It can be easily extended with threads, loops or whatever you need to set value to cache.

Call method once when Flask app started despite many Gunicorn workers

1 Answers1