Running Heroku background tasks with only 1 web dyno and 0 worker dynos

Question

I have a Python Flask app on Heroku that serves web pages but also allows certain tasks to be launched which I believe would be best structured as background tasks. As such, I've followed the Heroku rq tutorial to set up background tasks. My Procfile looks like this:

web: python app.py
worker: python worker.py

However, my processes are currently scaled web=1 worker=0. Given that this background process won't be run very often, it doesn't seem sensible to me to provision an entire dyno for it and then pay the $34/month for something that small.

Question:

• If I leave the worker process declared in my Procfile but keep the scaling at web=1 worker=0, will my queued processes eventually be run on my available web dyno? Or will the queued processes never run?
• If the queued processes will never run, is there another way to do this short of, for example, using twisted in my web app to run the tasks asynchronously?

Additional Information

worker.py looks like this:

import os
import redis
from rq import Worker, Queue, Connection

listen = ['high', 'default', 'low']

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

The logic in the main app that enqueues a process looks like this:

from rq import Queue
from worker import conn
q = Queue(connection=conn)

q.enqueue(myfunction, myargument)

Answer 1

Modify Procfile to look like this:

web: bin/web

Now create the bin directory, and create the file bin/web to look like this:

#!/bin/bash
python app.py &
python worker.py

Make sure you give this file the executable permission:

$ chmod +x bin/web

Answer 2

I am currently running both my web and backend scheduler in Heroku using only 1 dyno.

Idea is to provide one main python script for Heroku to start in 1 dyno. This script is used to start both the web server process(es) and the customer scheduler process(es). You can then define your jobs and add them to the custom scheduler.

APScheduler is used in my case.

This is what I did:

in Procfile:

 web: python run_app.py    #the main startup script

in the run_app.py:

# All the required imports
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor
from apscheduler.triggers.cron import CronTrigger
from run_housekeeping import run_housekeeping
from apscheduler.schedulers.background import BackgroundScheduler
import os

def run_web_script():
    # start the gunicorn server with custom configuration
    # You can also using app.run() if you want to use the flask built-in server -- be careful about the port
    os.system('gunicorn -c gunicorn.conf.py web.jobboard:app --debug')  

def start_scheduler():

     # define a background schedule 
     # Attention: you cannot use a blocking scheduler here as that will block the script from proceeding.
     scheduler = BackgroundScheduler()

     # define your job trigger
     hourse_keeping_trigger = CronTrigger(hour='12', minute='30')

     # add your job
     scheduler.add_job(func=run_housekeeping, trigger=hourse_keeping_trigger)

     # start the scheduler
     scheduler.start()


def run():
    start_scheduler()
    run_web_script()

if __name__ == '__main__':
    run()

I am also using 4 Worker processes for serving the web from Gunicorn -- which runs perfectly fine.

In gunicorn.conf.py:

loglevel = 'info'
errorlog = '-'
accesslog = '-'
workers = 4

You may want to checkout this project as an example: Zjobs@Github

Answer 3

You could use a process manager such as god or monit.

With god, you can set up your configuration like so

God.watch do |w|
  w.name = "app"
  w.start = "python app.py"
  w.keepalive
end

God.watch do |w|
  w.name = "worker"
  w.start = "python worker.py"
  w.keepalive
end

Then you put this in your Procfile

god -c path/to/config.god -D

By default, it automatically restarts the process if it crashes, and you can configure it to restart the app if memory usage gets too high. Check out the documentation.

Answer 4

To start and run a process in background:

Procfile:

run: python my_app.py

And later, execute:

heroku ps:scale run=1

Answer 5

You should take a look at Heroku Scheduler it will allow you to run a specific task on a scheduled interval such as every 10 minutes. If you already have your worker setup you could add:

heroku run worker