Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
4
votes
3 answers

Autodiscovering celery tasks

I have the following project tree (for testing purposes) and I am trying to understand how Celery is loading tasks. app ├── __init__.py ├── app.py ├── celery.py └── my_tasks ├── __init__.py └── tasks.py celery.py contains the following code…
Apostolos
  • 7,763
  • 17
  • 80
  • 150
4
votes
1 answer

Celery queues and Redis queues

I was looking at some tutorials for setting up Redis (message broker) + Celery for Django and I'm a little confused about how the queues work. The tutorial I followed was…
CodeGator
  • 73
  • 8
4
votes
1 answer

Celery lose worker

I use celery 4.4.0 version in my project(Ubuntu 18.04.2 LTS). When i raise Exception('too few functions in features to classify') , celery project lost worker and i get such logs: [2020-02-11 15:42:07,364] [ERROR] [Main ] Task handler raised…
Vadimcg
  • 125
  • 2
  • 9
4
votes
0 answers

amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg

I am trying to consume Celery task from a Rabbitmq queue. My tasks.py is below from celery import Celery app=Celery('tasks',broker ='amqp://localhost//') app.conf.task_default_queue = 'e' @app.task(name='hello') def hello(): …
newUser
  • 386
  • 5
  • 17
4
votes
1 answer

Celery chain performances

I wonder why celery chain is so slow comparing to an ad hoc solution. In the ad hoc solution I forward the task manually, the drawback is I cannot wait for the end of the chain. In the following code, the canvas solution takes 16 seconds and the ad…
ptitpoulpe
  • 684
  • 4
  • 17
4
votes
0 answers

Celery: Abort or revoke group in django celery

I'm using django celery. I know i can revoke task by using below code app.control.revoke(task_id, terminate=True) Can i use above code to remove group if the how can i get group id
Sandeep Prasad Kushwaha
  • 1,128
  • 11
  • 22
4
votes
3 answers

Django celery redis remove a specific periodic task from queue

There is a specific periodic task that needs to be removed from message queue. I am using the configuration of Redis and celery here. tasks.py @periodic_task(run_every=crontab(minute='*/6')) def task_abcd(): """ some operations here …
dipesh
  • 763
  • 2
  • 9
  • 27
4
votes
0 answers

redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe

I am running 13 celery Redis workers with a Redis broker When I execute the python producer code on my laptop, the code executes but I frequently get this error message on the screen I have tried to insert a sleep in the code as well and creat…
Chandra Sekhar Misra
  • 41
  • 2
4
votes
0 answers

Celery: Interact/Communicate with a running task

A related (albeit not identical) question appears here: Interact with celery ongoing task It's easy to start a task and get its unique ID: async_result = my_task.delay() task_id = async_result.task_id It's easy to broadcast a message that will…
Bernd Wechner
  • 1,854
  • 1
  • 15
  • 32
4
votes
1 answer

Starting multiple celery daemons automatically using Jenkins

I have an Ubuntu server set up with 5 different django sites running on it. These are used for testing, so each developer has their own site and database as well as one site for integrated code which is only updated when features are ready. …
mpdaugherty
  • 1,118
  • 8
  • 19
4
votes
1 answer

Python: Celery inspect

I found some related questions but nothing that describes exactly my problem. So I can inspect my queue with a code like that: from celery.task.control import inspect #inspect(['my_queue']), with a list instead of a str, should work! i =…
Varlor
  • 1,421
  • 3
  • 22
  • 46
4
votes
1 answer

Can you pass an object to a celery task by its memory id using ctypes?

I want to pass a object of my own class to a celery task. I am not using Django, this is my own custom class which is not serializable. After researching, I got the idea to pass the object memory id as an argument and then get the object from the id…
Nikolay Shindarov
  • 1,616
  • 2
  • 18
  • 25
4
votes
4 answers

How to implement Celery using Flask application factory pattern

I am having issues with implementing celery with python flask application factory app I have intend creating an instance of the Celery app from the app init file as below: from celery import Celery celery = Celery('myapp',…
Peter Ewanfo
  • 114
  • 1
  • 7
4
votes
2 answers

Celery tasks don't send email to admins on logger critical messages

My celery tasks don't send an email to application admins each time I call logger.critical. I'm building a Django application. The current configuration of my project, allows the admins of the application to receive an email each time a…
luistm
  • 1,027
  • 4
  • 18
  • 43
4
votes
3 answers

Celery + Redis backend: How to limit queue size?

Is there a way to limit queue size when I run Celery with Redis backend? Something like x-max-length in queue predeclare for rabbitmq
Maksym Polshcha
  • 18,030
  • 8
  • 52
  • 77
1 2 3
…
99 100