Celery beat not starting on Heroku

Question

I have an app that I've launched on Heroku, but the Celery beat process doesn't start when I start the server.

Procfile

web: gunicorn -w 4 connect.wsgi
celery: python manage.py celeryd -c 3 --beat

The worker can be seen to be started after the Heroku app is launched:

$ heroku ps

=== web (Free): gunicorn -w 4 connect.wsgi (1)
web.1: starting 2016/07/13 16:17:18 -0400 (~ 9s ago)

=== celery (Free): python manage.py celeryd -c 3 --beat (1)
celery.1: up 2016/07/13 16:17:25 -0400 (~ 2s ago)

However, in order to get the Celery beat process running, I have to explicitly start it in Heroku with:

heroku run python manage.py celerybeat

Celery beat launches fine locally. Is this a limitation of Heroku or am I doing something wrong?

score 7 · Answer 1 · answered Feb 24 '17 at 07:26

If your running on a free tier heroku with a single dyno then you best bet is to use honcho python clone of foreman a tool for managing Procfile-based applications. clone https://github.com/nickstenning/honcho, This will allow you to fork mutiple processes for your celery beat/workers. You will still be limited by heroku's free tier memory 512MB ram and dyno operating hours. So nothing too heavy good for quick dev and poc's

install honcho

pip install honcho

Ensure honcho is part of your requirement.txt

pip freeze > requirements.txt

Create a ProcfileHoncho store all your original Procfile content

ProcfileHoncho

web: gunicorn myDjangoApp.wsgi --log-file -
worker1: celery -A myDjangoApp beat -l info
worker2: celery -A myDjangoApp worker -l info

Procfile

web: honcho start -f ProcfileHoncho

ensure you load up your broker url via config vars and point to your free managed broker. Am sure you can find one free broker with a quick google search

git push heroku master
heroku logs -t

Check out the logs see if they are any errors. At this point you should be good to go.

score 4 · Accepted Answer · answered Jul 16 '16 at 13:39

4

Heroku only allow two free Dyno instances in one application if I'm not mistaken.

answered Jul 16 '16 at 13:39

Muhlen

176
2

That's too bad... It's weird that you can start a process in the terminal to get a third going. – Jared Goguen Jul 22 '16 at 20:11

score 2 · Answer 3 · answered Jul 21 '16 at 16:29

2

@Jared Goguen: Hi pal,

You probable need to scale up your workers in Heroku,

Deploying on Heroku

If you already created a Procfile above and attached the appropriate add-ons for the message broker and result store, all that’s left to do is push and scale your app:

git push heroku master

heroku ps:scale worker=1

Of course, at any time you can scale to any number of worker dynos. Now run a task just like you did locally:

heroku run python

>>> import tasks
>>> tasks.add.delay(1, 2)

You should see the task running in the application logs:

heroku logs -t -p worker

Source: Heroku Guides

answered Jul 21 '16 at 16:29

Rafael Aguilar

3,084
1
25
31

PS: I'm sorry about the poor format, SO gave me errors with quoted code. – Rafael Aguilar Jul 21 '16 at 16:30
Tried this before and ran into the "not enough dynos" issue, oh well. – Jared Goguen Jul 22 '16 at 20:13
As @Muhlen said in the other answer, you need to upgrade your account, of course you can start it from terminal, but the same dyno will be attending it (_not recommended_) – Rafael Aguilar Jul 23 '16 at 01:53

Celery beat not starting on Heroku

3 Answers3