Stack Overflow
Stack Overflow

Questions tagged [celeryd]

celeryd is the common name of a Celery worker node, running as a daemon.

Celery is a widely used asynchronous job/task queue processing engine, written in Python and available under the New BSD License. It can use a large variety of message brokers like RabbitMQ, Redis, MongoDB, CouchDB, Amazon SQS and others.

More info here: http://celery.github.com/celery/getting-started/introduction.html#overview

148 questions
6
votes
1 answer

Does the number of celeryd processes depend on the --concurrency setting?

We are running Celery behind Supervisor and start it with celeryd --events --loglevel=INFO --concurrency=2 This, however, creates a process graph that is up to three layers deep and contains up to 7 celeryd processes (Supervisor spawns one celeryd,…
Hannes Struß
  • 1,560
  • 2
  • 13
  • 20
5
votes
2 answers

How to configure Celery Daemon with Django

From what I can tell, there are two documents describing how to set up celery. There's "Running the worker as a daemon" and there's "First steps with Django". In the Django docs, it says: We also add the Django settings module as a configuration…
mlissner
  • 17,359
  • 18
  • 106
  • 169
5
votes
1 answer

Celery Result error "args must be a list or tuple"

I am running a Django website and have just gotten Celery to run, but I am getting confusing errors. Here is how the code is structured. In tests.py: from tasks import * from celery.result import AsyncResult project = Project.objects.create() #…
DoctorWizard
  • 303
  • 1
  • 3
  • 11
5
votes
1 answer

Celery: selectively clearing queue from Python tasks

Sometimes I have a situation where Celery queue builds up on accidental unnecessary tasks, clogging down the server. E.g. the code shoots up 20 000 tasks instead of 1. How one can inspect what Python tasks Celery queue contains and then get…
Mikko Ohtamaa
  • 82,057
  • 50
  • 264
  • 435
5
votes
0 answers

Configure Python Celery for Long Running Tasks

I have a setup where I run long idempotent tasks on AWS spot instances but I can't work out how to set up Celery to elegantly handle workers being killed mid task. At the moment if a worker is killed the task is marked as failed (WorkerLostError). I…
Aidan Kane
  • 3,856
  • 2
  • 25
  • 28
5
votes
1 answer

Best way to map a generated list to a task in celery

I am looking for some advice as to the best way to map a list generated from a task to another task in celery. Let's say I have a task called parse, which parses a PDF document and outputs a list of pages. Each page then needs to be individually…
chaimp
  • 16,897
  • 16
  • 53
  • 86
4
votes
1 answer

Celery worker variable sharing issues

I am using Python and celery in a project. In the project, I have two files: celeryconfig.py BROKER_URL = "amqp://guest:guest@localhost:5672//" CELERY_RESULT_BACKEND = "amqp" CELERY_IMPORTS = ("example",) CELERYD_CONCURRENCY = 2 and example.py from…
enderskill
  • 7,354
  • 3
  • 24
  • 23
4
votes
0 answers

amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg

I am trying to consume Celery task from a Rabbitmq queue. My tasks.py is below from celery import Celery app=Celery('tasks',broker ='amqp://localhost//') app.conf.task_default_queue = 'e' @app.task(name='hello') def hello(): …
newUser
  • 386
  • 5
  • 17
4
votes
3 answers

Celery + Redis backend: How to limit queue size?

Is there a way to limit queue size when I run Celery with Redis backend? Something like x-max-length in queue predeclare for rabbitmq
Maksym Polshcha
  • 18,030
  • 8
  • 52
  • 77
4
votes
1 answer

How to avoid SECRET_KEY error when starting Celery using Supervisor?

I have a Django 1.62 application running on Debian 7.8 with Nginx 1.2.1 as my proxy server and Gunicorn 19.1.1 as my application server. I've installed Celery 3.1.7 and RabbitMQ 2.8.4 to handle asynchronous tasks. I'm trying to manage my various…
Jim
  • 13,430
  • 26
  • 104
  • 155
4
votes
0 answers

Celery workers stop working with amazon sqs

I am using celery 3.1.9 with sqs. I have worker common_w running as daemon. It works with common queue on sqs. A worker unexpectedly stop processing tasks. No exceptions and errors. Last logs with option -l DEBUG: [2014-09-03 21:01:14,766:…
willfill
  • 294
  • 2
  • 9
4
votes
3 answers

Celeryd time constraint error

While running the status command I get the following error:- Am using rabbitmq as the messaging broker I am following this blog sudo /etc/init.d/celeryd status Error: No nodes replied within time constraint How can I Debug this error? I have also…
Akash Deshpande
  • 2,583
  • 10
  • 41
  • 82
4
votes
0 answers

Number of process and Currency and Pool Size in Celery

In celery 3.0, what are the relationship among Max concurrency, number of actually running processes and pool size ? I run celery with max-currency( -c ) set to one and three celeryd processes pops up. I do not know why.
onfire
  • 305
  • 2
  • 10
3
votes
2 answers

Celery worker which directory to start from

I need some help regarding Celery workers. I'm especially not able to understand where (which directory) does the celery worker command needs to be fired from and what is the concept behind it and some things around imports. So say I have the…
qre0ct
  • 5,680
  • 10
  • 50
  • 86
3
votes
3 answers

Django celery and celery-beat daemonizing script error

Im going to set periodic tasks with celery for what im trying to daemonize a django celery process in a test server using a script (referenced on the official website): Init-script: celery So, this is my celeryd file: # Names of nodes to start # …
Brian Ocampo
  • 1,345
  • 1
  • 12
  • 22
1 2
3
…
9 10