Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
4
votes
1 answer

Worker processes in Celery

I have a CPU intensive Celery task and within one of the task it can be further parallelized using joblib. By default, starting a worker with celery, will create a pool with number of max concurrency equal to number of CPUs/cores (which is 36 in my…
Coderji
  • 7,655
  • 5
  • 37
  • 51
4
votes
1 answer

Getting an error when writing a test for celery task

I wrote the following bare minimum unit test class for celery import pytest @pytest.fixture def celery_config(): return { "broker_url": "redis://localhost:6379/0", "result_backend": "redis://localhost:6379/0" …
not 0x12
  • 19,360
  • 22
  • 67
  • 133
4
votes
1 answer

Airflow: Celery workers too many MySQL connections

We are Running Airflow 1.10.1 with Celery. Facing multiple open connections. At the time DAGs kicks in - UI hangs for couple of minutes. Highlights: All nodes are BareMetal: CPU(s):40, MHz 2494.015, RAM 378G, 10Gb NIC - DB connections are not…
Eugene Bacal
  • 101
  • 5
4
votes
1 answer

VSCode debug Celery worker

After some frustrating days of running this, I'm needing to look at debugging a celery worker process in VSCode. This is following the suggested process in the Celery docs for creating a message handler, rather than pub/sub from the same…
Ross Halliday
  • 785
  • 6
  • 19
4
votes
4 answers

Get Image Field Absolute Path in Django Rest Framework - Non Request Flows

I've a periodic celery task which needs to store representation of a object in a specific json field. Here is the simplified model structure. Parent <-- ChildWrapper <-- Child Image So basically I've a 'ChildImage' model referring to 'ChildWrapper'…
Karthik RP
  • 1,028
  • 16
  • 26
4
votes
1 answer

Celery can't connect to local PostgreSQL from within Docker container

I have a Django-Celery-PostgreSQL project that should partially run in Docker. Having Redis and PostgreSQL servers running locally on the machine and using virtual environment for the project everything runs smoothly. But when I try to setup Docker…
SS_Rebelious
  • 1,274
  • 2
  • 19
  • 35
4
votes
1 answer

Logging Celery on Client/Producer

This is not a question about how to capture logging on celery workers. Is there any way to capture celery logging on a Producer. What I want is to capture every log that get generated by celery on the Producer when I call task.delay(...) or…
konoufo
  • 427
  • 5
  • 15
4
votes
4 answers

How to fix "Error Removing corrupted schedule file 'celerybeat-schedule': error('Bad magic number',)r"

I can run the celery beat normally using the command in the terminal, but when I go to run in aws, I get the following error: [2019-07-10 11:30:35,166: ERROR/MainProcess] Removing corrupted schedule file 'celerybeat-schedule': error('Bad magic…
user3358519
  • 125
  • 1
  • 8
4
votes
1 answer

How to add custom ID for Celery task in order to revoke task later?

I want to create delayed task and if there would be some conditions, I need to find that task and revoke it. This is how I create task: notify_before_departure.apply_async( args=(user.fcm_registration_token,…
belek
  • 907
  • 9
  • 35
4
votes
1 answer

Why I can't start celery with a path to an app

I have a tasks.py file I want to feed to Celery so it can run in the background. When I cdinto the directory, everything works properly celery -A tasks worker --loglevel=info However I'd like to run this in a bash script, so I have to specify the…
Arthur Attout
  • 2,701
  • 2
  • 26
  • 49
4
votes
1 answer

Celery worker receiving too many tasks

I have started my Celery worker like this: celery -A workerTasks worker --concurrency 1 --loglevel=info But i see that it is receiving multiple tasks: [2019-05-08 10:33:07,371: INFO/MainProcess] celery@2aaf46abfaed ready. [2019-05-08 10:33:07,372:…
Omroth
  • 883
  • 7
  • 24
4
votes
1 answer

Docker-compose: bind Celery to Postgres database

My Docker app runs with Flask as backend and Celery as an asyncronous task manager. Task results are then dumped in a SQLalchemy database owned by Postgres. However, I am not being able to make Celery interact with…
8-Bit Borges
  • 9,643
  • 29
  • 101
  • 198
4
votes
0 answers

Disable logging for a specific celery task

I have implemented logging for my celery app using the Python logging library. One current issue I am facing is that I have a lot of noise in my logs due to a beat that is running every second or so. In the grand scheme of things, I still want to…
stoneman_41
  • 370
  • 2
  • 12
4
votes
2 answers

Is it possible to wait until celery group done?

I am trying to do group tasks and wait until all the group subtasks finished then run the last task. But when I call task it calls group and last tasks but the last task finished before group finish. Is it possible to wait until all the tasks inside…
Zorig
  • 585
  • 1
  • 10
  • 26
4
votes
0 answers

ECS Airflow 1.10.2 performance issues. Operators and tasks take 10x longer

We moved to puckel/Airflow-1.10.2 to try and resolve a poor performance we've had in multiple environments. We are running on ECS Airflow 1.10.2 on AWS ECS. Interestingly, the CPU/mem never jump above 80%. The Airflow metadb stays very…
Seth
  • 446
  • 5
  • 10
1 2 3
…
99
100