How can to start tasks with celery-beat?

Question

Why i can't run a periodic tasks?

proj/settings.py

REDIS_HOST = 'localhost'
REDIS_PORT = '6379'
CELERY_BROKER_URL = 'redis://localhost:6379'
BROKER_URL = 'redis://' + REDIS_HOST + ':' + REDIS_PORT

CELERY_BEAT_SCHEDULE = {
    'task-first': {
        'task': 'app.tasks.one',
        'schedule': timedelta(seconds=1)
    },
    'task-second': {
        'task': 'app.tasks.two',
        'schedule': crontab(minute=0, hour='*/3,10-19')
    }
}

proj/celery.py

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')
app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

proj/__init.py__

from .celery import app as celery_app

__all__ = ['celery_app']

celery -A proj worker -l info

[2019-10-31 16:57:57,906: INFO/MainProcess] Connected to redis://localhost:6379//
[2019-10-31 16:57:57,912: INFO/MainProcess] mingle: searching for neighbors
[2019-10-31 16:57:58,927: INFO/MainProcess] mingle: all alone
[2019-10-31 16:57:58,951: INFO/MainProcess] celery@lexvel-MS-7A72 ready.

Tasks are found

celery -A proj beat -l info

Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 minutes (300s)
[2019-10-31 16:58:02,851: INFO/MainProcess] beat: Starting...

The celerybeat-shedule file is created. But besides these lines nothing more is displayed.

tasks

@task()
def one():
    print('start 1', datetime.now())
    driver = init_driver()
    parse(driver)
    driver.close()
    driver.quit()


@task()
def two():
    print('start 2', datetime.now())
    driver = init_driver()
    parse2(driver)
    driver.close()
    driver.quit()
    print('end 2', datetime.now())

score 10 · Answer 1 · edited Nov 12 '19 at 04:43

10

Celery beat command

celery -A proj worker -l info -B --scheduler django_celery_beat.schedulers:DatabaseScheduler

This command has used for start the celery beat.

Firstly add the django_celery_beat module in installed apps in settings file.

And then apply the django migrate command, this will create the tables in admin pannel.

After completing all the process like in celery file and create task in tasks.py.

you will apply the beat command as mentions in above.

proj/settings.py

INSTALLED_APPS = [

    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django_celery_beat',
]

REDIS_URL = "redis://localhost:6379/1"

CELERY_BROKER_URL=REDIS_URL

CELERY_RESULT_BACKEND=REDIS_URL

CELERY_ACCEPT_CONTENT = ['application/json']

CELERY_RESULT_SERIALIZER = 'json'

CELERY_TASK_SERIALIZER = 'json'

CELERY_BEAT_SCHEDULE = {

        'task-first': {
        'task': 'app.tasks.one',
        'schedule': timedelta(seconds=1)
       },
      'task-second': {
        'task': 'app.tasks.two',
        'schedule': crontab(minute=0, hour='*/3,10-19')
      }
}

proj/celery.py

from celery import Celery

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

app = Celery('proj')

app.config_from_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

proj/__init.py__

from .celery import app as celery_app

**__all__** = ['celery_app']

edited Nov 12 '19 at 04:43

Avi

1,424
1
11
32

answered Nov 01 '19 at 13:36

Manish Kumar

554
4
7

If I not write this line `BROKER_URL = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'` I get error `(403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.` – Nov 01 '19 at 13:55
This is confusing... Are you using Redis as broker, or RabbitMQ??? – DejanLekic Nov 04 '19 at 10:21
Here I have used the redis as a broker firstly you have to need install the redis-server, then you use the redis easily. But you could use RabbitMq as a broker. – Manish Kumar Nov 04 '19 at 12:06
@DejanLekic I use `redis` as broker and I not understand why get error related with `RabbitMQ` if I remove `BROKER_URL = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'` – Nov 05 '19 at 11:16
The command to start the beat also starts a worker. I suggest you do not do this, unless you are in the development environment. – DejanLekic Nov 05 '19 at 11:22
Could you please give me some reason why we don't use this command. – Manish Kumar Nov 05 '19 at 13:24

score 2 · Answer 2 · answered Nov 01 '19 at 13:41

2

When you ran your worker, did it say that app.tasks.one and app.tasks.two are registered tasks? If they do not appear there as registered tasks then your beat is scheduling tasks that can't be executed. - They will just wait in the queue, and eventually expire. Another way to check whether you have them registered is via celery -A proj.celeryapp inspect registered (change proj.celeryapp to whatever is the location of your Celery application).

answered Nov 01 '19 at 13:41

DejanLekic

18,787
4
46
77

When I run a `worker` I get list of my tasks `[tasks] (app.tasks.one; app.tasks.two) ` – Nov 01 '19 at 13:50
If I write `celery -A proj.celery inspect registered` I get error `Error: No nodes replied within time constraint.` and worker crashes with error `Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists. Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database. ` – Nov 01 '19 at 13:58
Are you getting this error while at least one worker is running? – DejanLekic Nov 01 '19 at 14:00
I was downgrade `kombu` to `4.6.3` version and run this command `celery -A proj.celery inspect registered`. On the output I get `-> celery@user: OK ( * app.tasks.one; * app.tasks.two) ` – Nov 01 '19 at 14:02

How can to start tasks with celery-beat?

2 Answers2

Celery beat command