Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
63
votes
6 answers

Running "unique" tasks with celery

I use celery to update RSS feeds in my news aggregation site. I use one @task for each feed, and things seem to work nicely. There's a detail that I'm not sure to handle well though: all feeds are updated once every minute with a @periodic_task, but…
Luper Rouch
  • 9,304
  • 7
  • 42
  • 56
62
votes
6 answers

Celery AttributeError: async error

I have RabbitMQ and Celery running locally on my Mac (OS/X 10.13.4), the following code works locally when I run add.delay(x,y): #!/usr/bin/env python from celery import Celery from celery.utils.log import get_task_logger logger =…
DYoung
  • 841
  • 1
  • 6
  • 11
62
votes
9 answers

How to dynamically add / remove periodic tasks to Celery (celerybeat)

If I have a function defined as follows: def add(x,y): return x+y Is there a way to dynamically add this function as a celery PeriodicTask and kick it off at runtime? I'd like to be able to do something like (pseudocode): some_unique_task_id =…
Jamie Forrest
  • 10,895
  • 6
  • 51
  • 68
61
votes
4 answers

Celery task that runs more tasks

I am using celerybeat to kick off a primary task that kicks of a number of secondary tasks. I have both tasks written already. Is there a way to easily do this? Does Celery allow for tasks to be run from within tasks? My example: @task def…
Mantas Vidutis
  • 16,376
  • 20
  • 76
  • 92
61
votes
3 answers

Cannot import name _uuid_generate_random in heroku django

I am working on a project which scans user gmail inbox and provides a report. I have deployed it in heroku with following specs: Language: Python 2.7 Framework: Django 1.8 Task scheduler: Celery (Rabbitmq-bigwig for broker url) Now when heroku…
akhi1
  • 1,322
  • 1
  • 16
  • 25
61
votes
8 answers

Django Celery - Cannot connect to amqp://guest@127.0.0.8000:5672//

I'm trying to set up Django-Celery. I'm going through the tutorial http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html when I run $ python manage.py celery worker --loglevel=info I get…
IdeoREX
  • 1,455
  • 5
  • 22
  • 39
60
votes
5 answers

RabbitMQ (beam.smp) and high CPU/memory load issue

I have a debian box running tasks with celery and rabbitmq for about a year. Recently I noticed tasks were not being processed so I logged into the system and noticed that celery could not connect to rabbitmq. I restarted rabbitmq-server and even…
marcin_koss
  • 5,763
  • 10
  • 46
  • 65
60
votes
2 answers

How to set up celery workers on separate machines?

I am new to celery.I know how to install and run one server but I need to distribute the task to multiple machines. My project uses celery to assign user requests passing to a web framework to different machines and then returns the result. I read…
Evan Gui
  • 765
  • 1
  • 5
  • 8
58
votes
7 answers

Celery with Amazon SQS

I want to use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. However there is not enough documentation for using it, so I cannot find how to configure SQS on Celery. Is there…
minhee
  • 5,688
  • 5
  • 43
  • 81
57
votes
14 answers

Django, ImportError: cannot import name Celery, possible circular import?

I went through this example here: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html All my tasks are in files called tasks.py. After updating celery and adding the file from the example django is throwing the following…
kev
  • 8,928
  • 14
  • 61
  • 103
57
votes
8 answers

Celery Flower Security in Production

I am looking to use Flower (https://github.com/mher/flower) to monitor my Celery tasks in place of the django-admin as reccomended in their docs (http://docs.celeryproject.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor).…
mh00h
  • 1,824
  • 3
  • 25
  • 45
57
votes
6 answers

Celery Worker Database Connection Pooling

I am using Celery standalone (not within Django). I am planning to have one worker task type running on multiple physical machines. The task does the following Accept an XML document. Transform it. Make multiple database reads and writes. I'm…
oneself
  • 38,641
  • 34
  • 96
  • 120
56
votes
2 answers

Retrieve task result by id in Celery

I am trying to retreive the result of a task which has completed. This works from proj.tasks import add res = add.delay(3,4) res.get() 7 res.status 'SUCCESS' res.id '0d4b36e3-a503-45e4-9125-cfec0a7dca30' But I want to run this from another…
Demetris
  • 2,981
  • 2
  • 25
  • 33
54
votes
5 answers

How to stop celery worker process

I have a Django project on an Ubuntu EC2 node, which I have been using to set up an asynchronous using Celery. I am following this along with the docs. I've been able to get a basic task working at the command line,…
user1592380
  • 34,265
  • 92
  • 284
  • 515
53
votes
3 answers

How can I recover unacknowledged AMQP messages from other channels than my connection's own?

It seems the longer I keep my rabbitmq server running, the more trouble I have with unacknowledged messages. I would love to requeue them. In fact there seems to be an amqp command to do this, but it only applies to the channel that your connection…
Will Olbrys
  • 957
  • 2
  • 9
  • 13
1 2
3
…
99 100