Stack Overflow
Stack Overflow

Questions tagged [celeryd]

celeryd is the common name of a Celery worker node, running as a daemon.

Celery is a widely used asynchronous job/task queue processing engine, written in Python and available under the New BSD License. It can use a large variety of message brokers like RabbitMQ, Redis, MongoDB, CouchDB, Amazon SQS and others.

More info here: http://celery.github.com/celery/getting-started/introduction.html#overview

148 questions
2
votes
1 answer

celery daemon: not found/celeryd error in configuration file

I was demonizing celery on Ubuntu by following the user guide and this question on SO. Briefly, Download the init-script celeryd from celery github repo. Save the configuration file from the user guide as celeryd.conf (to be distinguished form the…
azalea
  • 11,402
  • 3
  • 35
  • 46
2
votes
1 answer

Multiple queue for Celery daemon

I have a network of Celery servers and workers that are to be used for a high volume I/O task coming up. There are two queues, default and backlog, and each server has five workers. All servers are daemonized with a configuration much like the init…
Jason
  • 11,263
  • 21
  • 87
  • 181
2
votes
1 answer

Get the task_name from AsyncResult when submitting chains in celery

How am I supposed to retrieve a task's name when I've got its AsyncResult object and consequently its id? For example if I launch two of my tasks in a chain: >>> task_chain = (task_A.s() | task_B.s()) >>> async_result = task_chain.apply_async() I…
stratis
  • 7,750
  • 13
  • 53
  • 94
2
votes
1 answer

Celery not running in production

I am trying to run celery and celerybeat in production. In my current django app I am able to test and run using the commands "celery -A Gorgon worker" and "celery -A Gorgon beat -l debug --max-interval=10". Also, I am running it through a…
Sourabh Dev
  • 743
  • 11
  • 22
2
votes
0 answers

Can't get "retry" to work in Celery

Given a file myapp.py from celery import Celery celery = Celery("myapp") celery.config_from_object("celeryconfig") @celery.task(default_retry_delay=5 * 60, max_retries=12) def add(a, b): with open("try.txt", "a") as f: f.write("A…
H.D.
  • 4,168
  • 1
  • 18
  • 15
2
votes
2 answers

How can we see the threads in Celery?

I am trying to set a different number of concurrent threads in Celery using --concurrency=1 (based on Workers Guide). Anyway, when I run the daemon (celeryd ... --concurrency=1) I don't know how to see the number of threads that are running. Do they…
fedorqui
  • 275,237
  • 103
  • 548
  • 598
2
votes
1 answer

celery worker lost

Our deploy process restarts celery using a kill on all the celery proceses in the process list. Sometimes I see "worker lost" in the celery logs. If a task is running when this happens, will the task be re-run or will it be lost? We are using redis.…
Douglas Ferguson
  • 1,242
  • 2
  • 14
  • 25
2
votes
0 answers

Celerybeat with Mongo Connection Error

I'm trying to set up a celery task queue with celerybeat for periodic updates on a database. My celeryd service seems to working perfectly but celerybeat keeps giving me a Connection error: [Errno 111]: [2012-09-10 09:40:18,954: INFO/MainProcess]…
skarim
  • 95
  • 1
  • 7
2
votes
1 answer

Upstart - Django and Celeryd

I have my Ubuntu server setup so my Django project will bestarted by upstart like this: #!/bin/bash set -e LOGFILE=/var/log/gunicorn/foo.log LOGDIR=$(dirname $LOGFILE) NUM_WORKERS=3 # user/group to run as USER=django GROUP=django cd…
Harry
  • 13,091
  • 29
  • 107
  • 167
2
votes
3 answers

Questions Concerning Using Celery with Multiple Load-Balanced Django Application Servers

I'm interested in using Celery for an app I'm working on. It all seems pretty straight forward, but I'm a little confused about what I need to do if I have multiple load balanced application servers. All of the documentation assumes that the broker…
Lyle Pratt
  • 5,636
  • 4
  • 27
  • 28
2
votes
3 answers

how to track revoked tasks in across multiple celeryd processes

I have a reminder type app that schedules tasks in celery using the "eta" argument. If the parameters in the reminder object changes (e.g. time of reminder), then I revoke the task previously sent and queue a new task. I was wondering if there's any…
Eric Wang
  • 41
  • 3
1
vote
2 answers

How to configure and run remote celery worker correctly?

I'm new to celery and may be doing something wrong, but I already spent a lot of trying to figure out how to configure celery correctly. So, in my environment I have 2 remote servers; one is main (it has public IP address and most of the stuff…
Fedor
  • 1,392
  • 1
  • 17
  • 30
1
vote
1 answer

Celery's task_reject_on_worker_lost doesn't work with Redis as message broker

I'm currently using version 5.2.6 of Celery and version 6.2.6 of Redis. When I turn on the task_reject_on_worker_lost flag, I am expecting Celery to redeliver a task executed by a worker that died abruptly. However, trying this on Redis as message…
Jessica
  • 721
  • 1
  • 6
  • 13
1
vote
1 answer

Celery "Scheduler: Sending due task" logs printed as info level when expectation is debug

I'm still new to Python and Celery. I've setup Celery app to send all logs to Google Cloud Logging. It seems that part of the logs are sent as DEBUG, but some as INFO, causing me so much strain when trying to read them... Example logging…
dzh
  • 286
  • 1
  • 19
1
vote
0 answers

Running celery as daemon does not create PID file (no permission issue)

I am trying to run celery (worker) as a daemon / service in Ubuntu server. I've follow their documentation (https://docs.celeryproject.org/en/stable/userguide/daemonizing.html) However, when I start the daemon it says: celery multi v5.0.4…
Marvin
  • 79
  • 8
1 2 3
…
9 10