Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
4
votes
1 answer

How to run @classmethod as a Celery task in Django?

How to run @classmethod as a Celery task in Django? Here is example code: class Notification(TimeStampedModel): .... @classmethod def send_sms(cls, message, phone, user_obj=None, test=False): …
dev fan
  • 129
  • 7
4
votes
1 answer

thread safety in django with asynchronous tasks and redis

I have a django application that calls an asynchronous task on a queryset (using celery). The task takes the queryset and performs a whole bunch of operations that could potentially take a very long time based on the obects therein. Objects could be…
Bacon
  • 2,155
  • 4
  • 23
  • 31
4
votes
1 answer

Pass argument to custom celery task

I was just wondering what the proper way to pass an argument to a custom task decorator would be. For example, I found out I can subclass a celery task as follows: class MyTask(celery.Task): def __init__(self): # some custom stuff here …
Shaun
  • 480
  • 6
  • 10
4
votes
1 answer

Celery + Redis - Django blocks when triggering a task with delay

I have setup Celery on my Django project with Redis. The scheduled tasks are running without issues. The problems come when triggering an async task using the delay(). The execution stops and it's like is blocked in the loop of…
Fabio
  • 1,272
  • 3
  • 21
  • 41
4
votes
0 answers

How do I gracefully shutdown celery's flower?

When I send a SIGTERM, some times flower will die with this exception & its database does not contain the latest tasks. File "/home/devil/envs/celery/lib/python3.6/ site-packages/flower/app.py", line 68, in stop self.events.stop() File…
canadadry
  • 8,115
  • 12
  • 51
  • 68
4
votes
1 answer

Should I use a task queue (Celery), ayncio or neither for an API that polls other APIs?

I have written an API with Django which purpose is to operate as a bridge between a website back-end and external services we use, so that the website doesn't have to handle many requests to external APIs (CRM, calendar events, email providers…
Rémi Héneault
  • 383
  • 3
  • 12
4
votes
1 answer

Celery Periodic Tasks not running in Django

file structure proj/proj/ celery.py (and other files) /sitesettings/ tasks.py (and other files) celery.py app = Celery('mooncake',broker_url = 'amqp://') app.config_from_object('django.conf:settings',…
Tac
  • 355
  • 3
  • 12
4
votes
2 answers

Django-Celery Scheduling Daily Tasks on Windows Server

I have to schedule a task to run every Monday through Saturday. I downloaded django-celery to accomplish the task. I followed the tutorial on the Celery site and I used and looked at a few stack overflow posts, for example: here & here I did a mix…
L. P.
  • 165
  • 1
  • 19
4
votes
1 answer

How to reload a whole queryset from db in django?

I'm running periodic tasks with celery. One such task gets objects from db with by filter: pages = Page.objects.filter(active=True) initially(before starting celery itself) there are 10 such objects in db with active=True. The task executes…
Sreekanth Reddy Balne
  • 3,264
  • 1
  • 18
  • 44
4
votes
1 answer

Permission denied: '/code/celerybeat.pid'

I can't run Celery beat using Docker. celerybeat_1 | celery.platforms.LockFailed: [Errno 13] Permission denied: '/code/celerybeat.pid' docker service: celerybeat: <<: *django depends_on: - postgres - redis command:…
paul
  • 569
  • 9
  • 13
4
votes
0 answers

Celery executing task multiple time

I have a celery task as follows: @celery_app_site24x7.task(queue='site24x7') def createWebsiteMonitoring(**kwargs): """ Celery Task to create or update website Moniroting """ time.sleep(100) site24x7Instance =…
Kheshav Sewnundun
  • 1,236
  • 15
  • 37
4
votes
2 answers

Unicode Decode Error in Celery Trying to Read Results from Redis Queue

Trying to run a few simple tasks via celery. The header of the workerfile looks like from celery import Celery, group from time import sleep celery = Celery('workerprocess', broker='redis://localhost:6379/0',…
Della
  • 1,264
  • 2
  • 15
  • 32
4
votes
1 answer

How can I log from my python application to splunk, if I use celery as my task scheduler?

I have a python script running on a server, that should get executed once a day by the celery scheduler. I want to send my logs directly from the script to splunk. I am trying to use this splunk_handler library. If I run the splunk_handler without…
sidi7
  • 393
  • 4
  • 11
4
votes
2 answers

Dockerfile build from parent directory

I have a python application which uses celery workers which I would like to dockerise. Unfortunately, both python and Celery are using the same code base. Although they have to be separate containers for easier scaling. If I build them separately…
dben
  • 484
  • 1
  • 6
  • 21
4
votes
2 answers

How to start celery worker with variable used on worker initialization

Is there an option to pass variable to celery worker on start and use it inside worker on execution time? I'm writing server that will be responsible for machine learning training and evaluation. I would like to dynamically start new instance of…
Konrad
  • 952
  • 10
  • 25
1 2 3
…
99
100