Stack Overflow
Stack Overflow

Questions tagged [celery]

Celery is a distributed task queue framework for Python, used for asynchronous and parallel execution.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

For asynchronous tasks, Celery relies on a message broker (i.e. message transportation system). Among the options, RabbitMQ and Redis are the most stable and recommended ones. It's also possible to use a database, and there are also brokers in experimental phase, such as Amazon SQS and MongoDB.

Resources

8962 questions
39
votes
4 answers

Celery - run different workers on one server

I have 2 kind of tasks : Type1 - A few of high priority small tasks. Type2 - Lot of heavy tasks with lower priority. Initially i had simple configuration with default routing, no routing keys were used. It was not sufficient - sometimes all workers…
Andrew
  • 3,165
  • 4
  • 24
  • 29
39
votes
5 answers

How to inspect and cancel Celery tasks by task name

I'm using Celery (3.0.15) with Redis as a broker. Is there a straightforward way to query the number of tasks with a given name that exist in a Celery queue? And, as a followup, is there a way to cancel all tasks with a given name that exist in a…
Mzzzzzz
  • 4,770
  • 7
  • 30
  • 47
39
votes
2 answers

Running Scrapy spiders in a Celery task

I have a Django site where a scrape happens when a user requests it, and my code kicks off a Scrapy spider standalone script in a new process. Naturally, this isn't working with an increase of users. Something like this: class…
stryderjzw
  • 582
  • 1
  • 6
  • 9
38
votes
3 answers

How to put a rate limit on a celery queue?

I read this in the celery documentation for Task.rate_limit: Note that this is a per worker instance rate limit, and not a global rate limit. To enforce a global rate limit (e.g., for an API with a maximum number of requests per second), you must…
Vikash Singh
  • 13,213
  • 8
  • 40
  • 70
37
votes
6 answers

HEALTHCHECK of a Docker container running Celery tasks?

I know one of the ways to check health for Docker container is using the commmand HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1 But in case of workers there is no such URL to hit , How to check the container's health in that case ?
Always_Beginner
  • 2,546
  • 6
  • 25
  • 33
37
votes
6 answers

How are Django channels different than celery?

Recently I came to know about Django channels. Can somebody tell me the difference between channels and celery, as well as where to use celery and channels.
Vinay Singh
  • 458
  • 1
  • 4
  • 10
37
votes
5 answers

Run a Scrapy spider in a Celery Task

This is not working anymore, scrapy's API has changed. Now the documentation feature a way to "Run Scrapy from a script" but I get the ReactorNotRestartable error. My task: from celery import Task from twisted.internet import reactor from…
Juan Riaza
  • 1,618
  • 2
  • 16
  • 35
37
votes
2 answers

Python based asynchronous workflow modules : What is difference between celery workflow and luigi workflow?

I am using django as a web framework. I need a workflow engine that can do synchronous as well as asynchronous(batch tasks) chain of tasks. I found celery and luigi as batch processing workflow. My first question is what is the difference between…
user3343061
  • 478
  • 4
  • 7
36
votes
1 answer

Celery and Django simple example

Let's take a simple Django example. app/models.py from django.db import models from django.contrib.auth.models import User class UserProfile(models.Model): user = models.OneToOneField(User) token =…
Teodor Scorpan
  • 868
  • 1
  • 10
  • 20
36
votes
4 answers

Why does Celery NOT throw an Exception when the underlying task throws one

Celery doesn't seem to be handling exceptions properly. If I have task: def errorTest(): raise Exception() and then I call r = errorTest.delay() In [8]: r.result In [9]: r.state Out[9]: 'PENDING' And it will hang like this indefinitely. …
Kevin Meyer
  • 2,816
  • 4
  • 21
  • 33
36
votes
3 answers

How does a Celery worker consuming from multiple queues decide which to consume from first?

I am using Celery to perform asynchronous background tasks, with Redis as the backend. I'm interested in the behaviour of a Celery worker in the following situation: I am running a worker as a daemon using celeryd. This worker has been assigned two…
Jonathan Evans
  • 1,173
  • 2
  • 10
  • 22
35
votes
2 answers

Creating a Celery worker using node.js

Using node-celery, we can enable node to push Celery jobs to the task queue. How can we allow node to be a Celery worker and consume the queue?
Nyxynyx
  • 61,411
  • 155
  • 482
  • 830
35
votes
2 answers

Celery Logs into file

Can someone please help and tell me how to get the celery task debug details to a log file? I have a requirement to have the details of celery task logged into a .log file. Can you please make some suggestions on how this can be done without…
user2479840
  • 477
  • 1
  • 7
  • 14
35
votes
4 answers

Using Celery on processes and gevent in tasks at the same time

I'd like to use Celery as a queue for my tasks so my web app could enqueue a task, return a response and the task will be processed meanwhile / someday / ... I build a kind of API, so I don't know what sort of tasks will be there in advance - in…
Honza Javorek
  • 8,566
  • 8
  • 47
  • 66
34
votes
5 answers

How to Guarantee Message delivery with Celery?

I have a python application where I want to start doing more work in the background so that it will scale better as it gets busier. In the past I have used Celery for doing normal background tasks, and this has worked well. The only difference…
Ken Cochrane
  • 75,357
  • 9
  • 52
  • 60
1 2 3
…
99 100