celery-task is a building block of Celery distributed task queue system
celery-task is a building block of Celery distributed task queue system. Basically, it is a python class, but it can be created out of any callable by @celery.task decorator.
Questions tagged [celery-task]
727 questions
4
votes
2 answers
How to use asyncio and aioredis lock inside celery tasks?
Goal:
Possibility to run asyncio coroutines.
Correct celery behavior on exceptions and task retries.
Possibility to use aioredis lock.
So, how to run async tasks properly to achieve the goal?
What is RuntimeError: await wasn't used with future…
SH21
- 43
- 1
- 4
4
votes
1 answer
Celery task re-queued into broker on graceful shutdown when run locally but get lost in kubernetes despite same configs
I have a celery running in k8 pod. This is my manifest for celery
apiVersion: apps/v1
kind: Deployment
metadata:
name: celery
labels:
deployment: celery
spec:
replicas: 2
selector:
matchLabels:
pod: celery
template:
…
sap
- 234
- 5
- 16
4
votes
1 answer
Difference between send_task() and apply_async()
I'm not sure about the difference between apply_async() and send_task() in Celery library in Python, specially when using Tasks pools.
What I'm mainly looking for is having a centralized server with all tasks defined in there, and using some Redis…
bulDDoS
- 43
- 1
- 3
4
votes
1 answer
Celery chaining parallel tasks into chord
I'm trying to wrap my head around this asynchronous task processing setup. So far I've been looking at using Celery, but am not locked on to anything yet. Only requirement is I can use Redis as broker and distribute the tasks on multiple nodes.
…
Jakob Kristensen
- 1,647
- 3
- 12
- 22
4
votes
0 answers
Celery executing task multiple time
I have a celery task as follows:
@celery_app_site24x7.task(queue='site24x7')
def createWebsiteMonitoring(**kwargs):
""" Celery Task to create or update website Moniroting """
time.sleep(100)
site24x7Instance =…
Kheshav Sewnundun
- 1,236
- 15
- 37
4
votes
0 answers
Out of memory issue with celery and redis - need help tuning celery
I have a django 2 app that uploads 1 to many photographs (~3-4 MB per image), and uses face_recognition to find the face locations and encoding. The images and created thumbnails are saved to the file system and the data in a mysql database. The app…
user1045680
- 815
- 2
- 9
- 19
4
votes
3 answers
Possible race condition between Django post_save signal and celery task
In a django 2.0 app I have a model, called Document, that uploads and saves an image to the file system. That part works. I am performing some facial recognition on the image using https://github.com/ageitgey/face_recognition in a celery (v 4.2.1)…
user1045680
- 815
- 2
- 9
- 19
4
votes
1 answer
Celery worker ImportError: No module named 'project'
While I tried to start the worker I got a issue:
ImportError: No module named 'project'
Traceback (most recent call last):
File "/usr/local/bin/celery", line 11, in
sys.exit(main())
File…
William
- 3,724
- 9
- 43
- 76
4
votes
0 answers
Django Celery Group tasks executing only the first task
I have celery which has different tasks and has one queue.These tasks are not however all called at once depending on the request from the user the tasks called vary. So i have written a code that identifies which tasks to run and creates subtasks…
Anoop Antony
- 71
- 6
4
votes
1 answer
Celery - Cannot fetch task results
I'm using the Django ORM/Cache as the result backend for celery. I can see that django_celery_results_taskresult table is created in the database. After the tasks are finished the results are also inserted in the database which can be viewed from…
icyfire
- 1,170
- 8
- 21
4
votes
2 answers
Run celery periodic task for 1 hour, it it takes more then 1 hour, expire that task?
I have a periodic task for data purging which run once a day.
I want that task to run only for 1 hour. If the duration of processing that task is more than 1 hour, then expire that task. It will run again the next day for 1 hour.
I would like to do…
sachin gera
- 103
- 1
- 11
4
votes
2 answers
Best way to query Celery using AJAX from front-end to know if a created task is completed or not?
I am using Django with Celery + RabbitMQ to create video conversion tasks of videos being uploaded by the users. Now I know how to query celery to get the status. My question is where to save the task_id associated with each task, should I save it…
Chantz
- 5,883
- 10
- 56
- 79
4
votes
0 answers
Worker Lost Error after task accepted with exitcode 2
I am new to celery. I have added the backup script to celery using periodic_task. From the logs I see that "Task accepted: main" after immediately I see the below error in the logs.
[2017-09-21 06:01:00,257: ERROR/MainProcess] Process…
Anil Kumar Thotamalla
- 93
- 1
- 8
4
votes
0 answers
Cannot get State & Result Of Remote Celery Worker
My Architecture:
master-machine:
RabbitMQ 3.6.5 (broker)
Redis (backend)
Flower
Celery default worker
Webserver application
slave-machine1:
Celery default worker
slave-machine2:
Celery default worker
I'm triggering tasks via Webserver application…
ItayB
- 10,377
- 9
- 50
- 77
4
votes
1 answer
Sending bytes in a string to Redis in order to using it as parameter for Celery's task
I want to send bytes in a string to Celery's task using Redis as broker, but I'm receiving error shown below:
[2017-06-17 21:27:13,826] ERROR in app: Exception on /endpoint_method [POST]
Traceback (most recent call last):
File…
dawlib
- 237
- 2
- 13