Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
53
votes
4 answers

AttributeError: 'Flask' object has no attribute 'user_options'

I am trying to setup this basic example from the following doc: http://flask.pocoo.org/docs/patterns/celery/ But so far I keep getting the below error: AttributeError: 'Flask' object has no attribute 'user_options' I am using celery 3.1.15. from…
Ravdeep
  • 730
  • 1
  • 8
  • 13
52
votes
9 answers

Find out whether celery task exists

Is it possible to find out whether a task with a certain task id exists? When I try to get the status, I will always get pending. >>> AsyncResult('...').status 'PENDING' I want to know whether a given task id is a real celery task id and not a…
dominik
  • 5,745
  • 6
  • 34
  • 45
52
votes
3 answers

Python task queue alternatives and frameworks

There seem to be different implementations of task/job queues for Python 3: Celery, popular but apparently unmaintained and stale; RQ, of which I have little information; TaskTiger, similarly to RQ I know little about it; Huey , similarly to RQ I…
Jens
  • 8,423
  • 9
  • 58
  • 78
52
votes
2 answers

Why would running scheduled tasks with Celery be preferable over crontab?

Considering Celery is already a part of the stack to run task queues (i.e. it is not being added just for running crons, that seems an overkill IMHO ). How can its "periodic tasks" feature be beneficial as a replacement for crontab ? Specifically…
DhruvPathak
  • 42,059
  • 16
  • 116
  • 175
51
votes
9 answers

How do I restart celery workers gracefully?

While issuing a new build to update code in workers how do I restart celery workers gracefully? Edit: What I intend to do is to something like this. Worker is running, probably uploading a 100 MB file to S3 A new build comes Worker code has…
Quintin Par
  • 15,862
  • 27
  • 93
  • 146
51
votes
2 answers

Celery unable to use redis

Trying to start Celery first time but issues error as below, i have installed redis and its starting fine , but still somehow django seems to have issues with it , File "", line 848, in exec_module File…
Atif Shafi
  • 954
  • 1
  • 11
  • 26
50
votes
2 answers

How do I use the @shared_task decorator for class based tasks?

As seen in the documentation the @shared_task decorator lets you create tasks without having any concrete app instance. The given examples show how to decorate a function based task. Can you help me understand how to decorate a class based task?
Juan Riaza
  • 1,618
  • 2
  • 16
  • 35
50
votes
4 answers

How to use Flask-SQLAlchemy in a Celery task

I recently switch to Celery 3.0. Before that I was using Flask-Celery in order to integrate Celery with Flask. Although it had many issues like hiding some powerful Celery functionalities but it allowed me to use the full context of Flask app and…
PanosJee
  • 3,866
  • 6
  • 36
  • 49
48
votes
13 answers

How to run celery on windows?

How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker?
nicks
  • 2,161
  • 8
  • 49
  • 101
48
votes
3 answers

Using mock to patch a celery task in Django unit tests

I'm trying to use the python mock library to patch a Celery task that is run when a model is saved in my django app, to see that it's being called correctly. Basically, the task is defined inside myapp.tasks, and is imported at the top of my…
Emil
  • 1,949
  • 2
  • 16
  • 25
48
votes
3 answers

How to send periodic tasks to specific queue in Celery

By default Celery send all tasks to 'celery' queue, but you can change this behavior by adding extra parameter: @task(queue='celery_periodic') def recalc_last_hour(): log.debug('sending new task') recalc_hour.delay(datetime(2013, 1, 1, 2))…
Artem Mezhenin
  • 5,539
  • 6
  • 32
  • 51
48
votes
3 answers

How do you run a worker with AWS Elastic Beanstalk?

I am launching a Django application on AWS Elastic Beanstalk. I'd like to run a background task or worker in order to run celery. I can not find if it is possible or not. If yes how could it be achieved? Here is what I am doing right now, but this…
Maxime P
  • 775
  • 1
  • 9
  • 13
48
votes
4 answers

Celery: How to ignore task result in chord or chain?

I'm using celery, I have several tasks which needed to be executed in order. For example I have this task: @celery.task def tprint(word): print word And I want to do something like this: >>> chain(tprint.s('a') | tprint.s('b'))() Then I get…
lxyu
  • 2,661
  • 5
  • 23
  • 29
47
votes
9 answers

What's the equivalent of Python's Celery project for Java?

I am trying to find an equivalent of Celery project for Java environment, I have looked at Spring Batch, but are there any better alternatives for distributed task queues. Thanks.
Zakiullah Khan
  • 1,445
  • 2
  • 15
  • 26
47
votes
4 answers

How to make a celery task fail from within the task?

Under some conditions, I want to make a celery task fail from within that task. I tried the following: from celery.task import task from celery import states @task() def run_simulation(): if some_condition: …
Meilo
  • 3,378
  • 3
  • 28
  • 34
1 2 3
…
99 100