Single apscheduler instance in Flask application

Question

Setup:

Flask application running in Apache's httpd via wsgi
Single wsgi process with 25 threads: WSGIDaemonProcess myapp threads=25
apscheduler to run jobs (send emails)
RethinkDB as the backend for the job store

I'm trying to prevent apscheduler from running the same job multiple times by preventing multiple instances of apscheduler from starting. Currently I'm using the following code to ensure the scheduler is only started once:

    if 'SCHEDULER' not in app.config or app.config['SCHEDULER'] is None:
        logger.info("Configuring scheduler")
        app.config['SCHEDULER'] = scheduler.configure()

However, when I look at my logs, I see the scheduler being started twice:

[07:07:56.796001 pid 24778 INFO] main.py 57:Configuring scheduler
[07:07:56.807977 pid 24778 INFO] base.py 132:Scheduler started
[07:07:56.812253 pid 24778 DEBUG] base.py 795:Looking for jobs to run
[07:07:56.818019 pid 24778 DEBUG] base.py 840:Next wakeup is due at-10-14 11:30:00+00:00 (in 1323.187678 seconds)
[07:07:57.919869 pid 24777 INFO] main.py 57:Configuring scheduler
[07:07:57.930654 pid 24777 INFO] base.py 132:Scheduler started
[07:07:57.935212 pid 24777 DEBUG] base.py 795:Looking for jobs to run
[07:07:57.939795 pid 24777 DEBUG] base.py 840:Next wakeup is due at-10-14 11:30:00+00:00 (in 1322.064753 seconds)

As can be seen by the pid, there are two processes that are being started somewhere/somehow. How can I prevent this? Where is this configuration in httpd?

Say I did want two processes running, I could use flock to prevent apscheduler from starting twice. However, this won't work because the process that does NOT start apscheduler won't be able to add/remove jobs because app.config['SCHEDULER'] set for that process to use.

What is the best way to configure/setup a Flask web app with multiple processes that can add/remove jobs, and yet prevent the scheduler from running the job multiple times?

I was able to get httpd to only start once process by changing the mpm_event `StartServers` directive to 1. But this doesn't mean it won't start more processes... — wspeirs, Oct 14 '15 at 14:47

score 4 · Accepted Answer · answered Oct 21 '15 at 13:13

I finally settled on using a file-based lock to ensure that the task doesn't run twice:

def get_lock(name):
    fd = open('/tmp/' + name, 'w')

    try:
        flock(fd, LOCK_EX | LOCK_NB)  # open for exclusive locking
        return fd
    except IOError as e:
        logger.warn('Could not get the lock for ' + str(name))
        fd.close()
        return None


def release_lock(fd):
    sleep(2)  # extend the time a bit longer in the hopes that it blocks the other proc
    flock(fd, LOCK_UN)
    fd.close()

It's a bit of a hack, but seems to be working...

Single apscheduler instance in Flask application

1 Answers1