Stack Overflow
Stack Overflow

Questions tagged [celery-task]

celery-task is a building block of Celery distributed task queue system

celery-task is a building block of Celery distributed task queue system. Basically, it is a python class, but it can be created out of any callable by @celery.task decorator.

727 questions
14
votes
4 answers

Get the name of celery worker from inside a celery task?

I'd like a celery task to be able to get the name of the worker executing it, for logging purposes. I need to handle this from within the task, rather than querying the broker directly. Is there a way to do this? I'm using celery with RabbitMQ,…
imichaeldotorg
  • 459
  • 1
  • 4
  • 20
13
votes
1 answer

How to solve the a celery worker configuration with keyword argument namespace='"CELERY"error in the celery file

I have a project name called ShippingApp and I followed the steps to set up the celery worker. I am using celery 3.1.26.post2 with python3.7 and when I want to start the Celery Worker I am getting the error below: E:\ShippingApp>celery -A…
Mohamed Abdillah
  • 379
  • 1
  • 3
  • 16
13
votes
4 answers

Django Celery results set task id to something human readable?

After many days I have a working celery and celery beat task list, and the results are stored using django_celery_results. However when I look at the table record, it hasn't got any useful information in there. is it possible to set the task id to…
AlexW
  • 2,843
  • 12
  • 74
  • 156
13
votes
2 answers

worker does not consume tasks after celery add_consumer is called

I would like to leverage Celery (with RabbitMQ as backend MQ) to execute tasks of varying flavors via different Queues. One requirement is that consumption (by the workers) from a particular Queue should have the capability to be paused and…
pangyuteng
  • 1,749
  • 14
  • 29
12
votes
1 answer

Celery + SQS is receiving same task twice, with same task id at same time

using celery with SQS in flask app but celery is receiving same task twice with same task id at same time, running worker like this, celery worker -A app.jobs.run -l info --pidfile=/var/run/celery/celery.pid --logfile=/var/log/celery/celery.log…
ssnitish
  • 155
  • 9
12
votes
3 answers

Prioritizing queues among multiple queues in celery?

We are using celery for our asynchronous background tasks and we have 2 queues for different priority tasks. We have 2 cluster of nodes serving them separately. Things are working well as expected. Question: We get mostly low priority tasks. For…
arunk2
  • 2,246
  • 3
  • 23
  • 35
12
votes
4 answers

When I use Django Celery apply_async with eta, it does the job immediately

i looked at celery documentation and trying something from it but it not work like the example. maybe i'm wrong at some point, please give me some pointer if i'm wrong about the following code in views.py i have something like this: class…
knightzoid
  • 147
  • 1
  • 8
11
votes
1 answer

In celery, what is the appropriate way to pass contextual metadata from sender process to worker when a task is enqueued?

When any celery task is enqueued I want to add contextual metadata the worker will be able to use. The following code example works but I would like to have an appropriate celery-style solution. from celery.signals import before_task_publish,…
jrobichaud
  • 1,272
  • 1
  • 11
  • 23
11
votes
2 answers

Test if a celery task is still being processed

How can I test if a task (task_id) is still processed in celery? I have the following scenario: Start a task in a Django view Store the BaseAsyncResult in the session Shutdown the celery daemon (hard) so the task is not processed anymore Check if…
Reto Aebersold
  • 16,306
  • 5
  • 55
  • 74
11
votes
1 answer

Celery chain not working with batches

At first glance I liked very much the "Batches" feature in Celery because I need to group an amount of IDs before calling an API (otherwise I may be kicked out). Unfortunately, when testing a little bit, batch tasks don't seem to play well with the…
ninja.user
  • 382
  • 3
  • 10
11
votes
3 answers

Celery tasks not throwing exception in Django Tests

I have a couple of celery tasks that are included in my Django tests. Unfortunately exceptions are not thrown when tasks are invoked via .delay(). I am setting CELERY_ALWAYS_EAGER to True. tasks.py import celeryapp as app @app.task() def…
kev
  • 8,928
  • 14
  • 61
  • 103
11
votes
1 answer

Celery chain breaks if one of the tasks fail

What is the point of Celery chain if the whole chain breaks if one of the tasks fail?!! I have this Celery chain: res = chain(workme.s ( y=1111 ), workme2.s( 2222 ), workme3.s( 3333 ),)() And I made workme2 fails with retries like…
securecurve
  • 5,589
  • 5
  • 45
  • 80
10
votes
4 answers

KeyError Received unregistered task of type '' on celery while task is registered

I'm a bit new in celery configs. I have a task named myapp.tasks.my_task for example. I can see myapp.tasks.my_task in registered tasks of celery when I use celery inspect registered. doesn't it mean that the task is successfully registered? why it…
Ashkan Khademian
  • 307
  • 3
  • 12
10
votes
3 answers

Flask Celery task locking

I am using Flask with Celery and I am trying to lock a specific task so that it can only be run one at a time. In the celery docs it gives a example of doing this Celery docs, Ensuring a task is only executed one at a time. This example that was…
Max Powers
  • 1,119
  • 4
  • 23
  • 54
10
votes
1 answer

Size limit on Celery task arguments?

We have a Celery task that requires a Pandas dataframe as an input. The dataframe is first serialized to JSON and then passed as an argument into the task. The dataframes can have around 35 thousand entries, which results in a JSON dictionary…
biddlesby
  • 453
  • 4
  • 10
1 2
3
…
48 49