Stack Overflow
Stack Overflow

Questions tagged [celeryd]

celeryd is the common name of a Celery worker node, running as a daemon.

Celery is a widely used asynchronous job/task queue processing engine, written in Python and available under the New BSD License. It can use a large variety of message brokers like RabbitMQ, Redis, MongoDB, CouchDB, Amazon SQS and others.

More info here: http://celery.github.com/celery/getting-started/introduction.html#overview

148 questions
3
votes
1 answer

celery daemon production local config file without django

I am newbie to Celery. I create a project as per instruction provided by the celery4.1 docs.Below is my project folder and files: mycelery | test_celery | celery_app.py …
Manish Yadav
  • 435
  • 2
  • 8
  • 24
3
votes
0 answers

What is the optimal number of celery workers per core?

The celery docs don't have too much information about the optimal number of celery workers to use per machine. I believe by default celery creates one worker per machine core. I know from experimentation that on a single core machine, starting more…
jeffrey
  • 3,196
  • 7
  • 26
  • 44
3
votes
1 answer

Celery workers not filling concurrency slots

I have one worker with a concurrency of 4. I see 4 processes started in flower and everything looks good. If I do this in a shell then I see the 4 workers take the tasks and the rest are reserved and it processes 4 at a time until the queue is…
Mingle
  • 846
  • 6
  • 13
3
votes
0 answers

How to force celery to consume the next task if there are idle processes?

I have celery setup to fetch tasks from RabbitMQ and things are working as excepted, but I've noticed the following behavior (T: task, P: process): --> Fetch first batch of messages (6 tasks) from broker <-- messages are received. Start them -->…
andersonvom
  • 11,701
  • 4
  • 35
  • 40
3
votes
0 answers

Celery routing to multiple tasks rather than hosts

I am working on porting an application I wrote from Golang (using redis) to Python and I would love to use Celery to accomplish my task queuing, but I have a question regarding routing... My application receives "events" via REST POSTs, where each…
juanvilla
  • 196
  • 4
3
votes
0 answers

Celery work process consumes 100% CPU after running for several days

I submitted one issue to Celery project, see the details here: Celery worker process CPU 100% After running Celery worker process for some days, we found that the Python process consumes the 100% CPU load. I used the strace to dump the runtime…
Wyatt Fang
  • 39
  • 4
3
votes
1 answer

Celeryd launching too many processes

How do you ensure celeryd only runs as a single process? When I run manage.py celeryd --concurrency=1 and then ps aux | grep celery I see 3 instances running: www-data 8609 0.0 0.0 20744 1572 ? S 13:42 0:00 python manage.py celeryd…
Cerin
  • 60,957
  • 96
  • 316
  • 522
3
votes
1 answer

Celery single task persistent data

Lets say a single task is enough for a machine to stay very busy for a few minutes. I want to get the result of the task, then depending on the result, have the worker perform the same task again. The question I cannot find an answer to is this:…
phil0stine
  • 303
  • 1
  • 13
2
votes
1 answer

TransactionManagementError with Django Celeryd / django-celery with sqlite backend

Using Django for a small app that executes jobs in the background (mainly cron style, but also user initiated as well).     disclaimer: this is my first encounter with celeryd, setup so far is from docs and examples I'm using Django 1.3, celeryd…
Justin
  • 4,434
  • 4
  • 28
  • 37
2
votes
1 answer

Celery in Django (RabbitMQ vs. Django Database)

I am trying to set up Django with Celery so I can send bulk emails in the background. I am a little confused about how the different components play into Celery. Do I need to use RabbitMQ? Can I just "django-kombu" to run Celery?…
saul.shanabrook
  • 3,068
  • 3
  • 31
  • 49
2
votes
0 answers

Errors on Celery Daemon Startup

I need to setup celery on production so I'am daemonizing celery in a Ubuntu machine. I'm getting errors on service startup but I cannot figure out how to solve them. Here is my /etc/default/celeryd…
Stack
  • 1,028
  • 2
  • 10
  • 31
2
votes
1 answer

Continously running Celery Task in Django

I've a Django app which should constantly listen for messages from Kafka and then send them via WebSocket to client. Problem is how to setup constant listener. For future scalability we decided to bring Celery in project to manage these issues with…
quantagonista
  • 23
  • 6
2
votes
1 answer

Celery: how to assign some specific number of workers to different queues?

I have a celery app consuming from large amount of queues with gevent concurrency. For each queue I need to have some guaranteed number of workers, but the number is different for each queue. For example: qA must have 10 qB must have 25 qN must…
Maksym Polshcha
  • 18,030
  • 8
  • 52
  • 77
2
votes
0 answers

Will celeryd restart drop currently running tasks?

If I have two worker processes doing long-time operations. If I use /etc/init.d/celeryd restart as in the official document to restart them when they are in the middle of processing tasks, what happens then? Will they wait till they finish the tasks…
Borg Lin
  • 123
  • 1
  • 9
2
votes
1 answer

worker: Warm shutdown (MainProcess) after task started

Setup: Celery 3.1, broker=RabbitMQ, backend=Redis. Scenario: Having a task is state=STARTED (running) my worker is being restarted. I'm getting worker: Warm shutdown (MainProcess) message (stdout). The worker successfully restarted but the task is…
ItayB
  • 10,377
  • 9
  • 50
  • 77
1 2 3
…
9 10