Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
47
votes
11 answers

How do I enable remote celery debugging in PyCharm?

I'm trying to find some instructions on how to enable PyCharm debugging within my celery processes on a remote machine. The remote machine is running Ubuntu 14.04. I am running PyCharm 4.x. I've seen some other information that alludes others have…
Ryan
  • 993
  • 1
  • 9
  • 17
47
votes
8 answers

Celery auto reload on ANY changes

I could make celery reload itself automatically when there is changes on modules in CELERY_IMPORTS in settings.py. I tried to give mother modules to detect changes even on child modules but it did not detect changes in child modules. That make me…
Ahmet DAL
  • 4,445
  • 9
  • 47
  • 71
47
votes
1 answer

Celery: is there a way to write custom JSON Encoder/Decoder?

I have some objects I want to send to celery tasks on my application. Those objects are obviously not json serializable using the default json library. Is there a way to make celery serialize/de-serialize those objects with custom JSON…
Rafael S. Calsaverini
  • 13,582
  • 19
  • 75
  • 132
47
votes
7 answers

Dealing with duplicate primary keys on insert in SQLAlchemy (declarative style)

My application is using a scoped session and the declarative style of SQLALchemy. It's a web-app and a lot of the DB insertions are executed by Celery, a task scheduler. Typically, when deciding to insert an object, my code might do something along…
Edwardr
  • 2,906
  • 3
  • 27
  • 30
45
votes
2 answers

Why do CELERY_ROUTES have both a "queue" and a "routing_key"?

My understanding of AMQP is that messages only have the following components: The message body The routing key The exchange Queues are attached to exchanges. Messages can't have any knowledge of queues. They just post to an exchange, and then…
Adam Easterling
  • 2,266
  • 1
  • 21
  • 21
45
votes
2 answers

How to keep multiple independent celery queues?

I'm trying to keep multiple celery queues with different tasks and workers in the same redis database. Really just a convenience issue of only wanting one redis server rather than two on my machine. I followed the celery tutorial docs verbatim, as…
jwoww
  • 991
  • 1
  • 11
  • 19
44
votes
1 answer

How can I use PyCharm to locally debug a Celery worker?

I have an existing Django project with a virtualenv. After activating the venv, I can run Celery just with the command celery. This works on Windows, OS X and Linux. I wanted to try PyCharm on Windows, and I'm able to get it to run my Django server…
Shaggy Frog
  • 27,575
  • 16
  • 91
  • 128
44
votes
1 answer

what's difference between celeryd, celery worker, celerybeat?

What is the difference between these? celeryd celery worker celerybeat I'm trying to setup celery + supervisor and some conf file on the net has more than one while others have only one.
eugene
  • 39,839
  • 68
  • 255
  • 489
43
votes
4 answers

What are the consequences of disabling gossip, mingle and heartbeat for celery workers?

What are the implications of disabling gossip, mingle, and heartbeat on my celery workers? In order to reduce the number of messages sent to CloudAMQP to stay within the free plan, I decided to follow these recommendations. I therefore used the…
nbeuchat
  • 6,575
  • 5
  • 36
  • 50
42
votes
3 answers

How to disallow pickle serialization in celery

Celery defaults to using pickle as its serialization method for tasks. As noted in the FAQ, this represents a security hole. Celery allows you to configure how tasks get serialized using the CELERY_TASK_SERIALIZER configuration parameter. But…
Leopd
  • 41,333
  • 31
  • 129
  • 167
41
votes
1 answer

Python+Celery: Chaining jobs?

The Celery documentation suggests that it's a bad idea to have tasks wait on the results of other tasks… But the suggested solution (see “good” heading) leaves a something to be desired. Specifically, there's no clear way of getting the subtask's…
David Wolever
  • 148,955
  • 89
  • 346
  • 502
41
votes
5 answers

Celery - How to send task from remote machine?

We have a server running celery workers and a Redis queue. The tasks are defined on that server. I need to be able to call these tasks from a remote machine. I know that it is done using send_task but I still haven't figured out HOW? How do I tell…
user1102018
  • 4,369
  • 6
  • 26
  • 33
40
votes
3 answers

Celery Get List Of Registered Tasks

Is there a way to get a list of registered tasks? I tried: celery_app.tasks.keys() Which only returns built in Celery tasks like celery.chord, celery.chain etc.
Richard Knop
  • 81,041
  • 149
  • 392
  • 552
40
votes
4 answers

In-Memory broker for celery unit tests

I have a REST API written in Django, with and endpoint that queues a celery task when posting to it. The response contains the task id which I'd like to use to test that the task is created and get the result. So, I'd like to do something like: def…
Facundo Olano
  • 2,492
  • 2
  • 26
  • 32
39
votes
1 answer

Retrieve a task result object, given a `task_id` in Celery

I store the task_id from an celery.result.AsyncResult in a database and relate it to the item that the task affects. This allows me to perform a query to retrieve all the task_ids of tasks that relate to a specific item. So after retrieving the…
bradley.ayers
  • 37,165
  • 14
  • 93
  • 99
1 2 3
…
99 100