celery takes too long time to write result to rabbitmq

Asked Dec 04 '14 at 03:24

Active Dec 04 '14 at 03:24

Viewed 320 times

Recently, I started celery beat to run a task periodically. The task will take about 2 minutes. The beat interval is 3 minutes. The back end use rabbitmq.

However, the totally elapsed time of a task become nearly 20 minutes. It looks so strange! After some work, I found that the extra time consumed by sending task result to rabbitmq. It is awesome! Why?

And the celery worker will use another 5 or 7 minutes to receive the next task. I do not know what the worker are doing in this period.

Anyone could help to explain them?

asked Dec 04 '14 at 03:24

Jacky1205

3,273
3
22
44

What the task receives from RabbitMQ/writes back? What is the value of `ignore_results`? What is the backend storage? – Krzysztof Szularz Dec 04 '14 at 09:37
The result of the task is a simple object of dict type. The size is less than 32 bytes. `ignore_result` does not enabled. And the backend storage is Rabbitmq. – Jacky1205 Dec 04 '14 at 09:52
And a funny fact is that, it become normal when I decrease the amount of beat interval, to set to 150 seconds, which is almost equal to the whole run time of the task. – Jacky1205 Dec 04 '14 at 09:54
How many tasks there are? RabbitMQ result backend creates new queue *per task*. – Krzysztof Szularz Dec 05 '14 at 08:55
About 260. It is not too many. – Jacky1205 Dec 05 '14 at 15:15

celery takes too long time to write result to rabbitmq

0 Answers0