How to measure Celery task duration

Question

Is there an easy way to measure the time it takes for a Celery task to be processed and successfully finished? There is a similar question but the first answer is ugly and unfortunately the signals are not called for me as second answer suggested.

Flower UI has a Runtime column but it's API doesn't allow to get batch of tasks.

Are you looking to measure the time from creating the task to getting the result of the task or are you just wanting to time the length of the actual task? — Ben McAlindin, Dec 31 '15 at 18:31
@BenMcAlindin, are they different? Basically, I want to measure the time the task is created until the result is back. — Sam R., Dec 31 '15 at 18:33
For one task it shouldn't be (except for some small overhead). If you have many tasks being queued, often from many users, this could can cause some tasks to be blocked and take longer to return results. — Ben McAlindin, Dec 31 '15 at 18:39
@BenMcAlindin, that makes sense. So it'll be the time that the task has been created until the result is back. Full round-trip. — Sam R., Dec 31 '15 at 18:42
A possible trick is to use log messages: a message at begin and a message at the end: then you can see the duration of each task (consider the task id in log messages). — Laurent LAPORTE, Dec 31 '15 at 20:23
Possible duplicate of [Measuring Celery task execution time](https://stackoverflow.com/questions/19481470/measuring-celery-task-execution-time) — georgexsh, Sep 28 '17 at 17:34

score 3 · Answer 1 · answered Aug 03 '17 at 16:59

Task events can be used to get the run time duration of a task. For more details, please refer this link. Task-sent, task-received, task-started, task-succeeded, task-failed, task-rejected, task-revoked, task-retried are the task events supported in celery. To measure the task duration, 'task-succeeded' event could be used. Please find below the code snippet.

    from celery import Celery
    redis = Redis(host='workerdb', port=6379, db=0)

    def my_monitor():
        app = Celery('vwadaptor', broker='redis://workerdb:6379/0',backend='redis://workerdb:6379/0')
        state = app.events.State()

        def announce_task_succeeded(event):
            state.event(event)
            task = state.tasks.get(event['uuid'])
            print "task runtime: ", task.runtime

        with app.connection() as connection:
            recv = app.events.Receiver(connection, handlers={
                    'task-succeeded': announce_task_succeeded, 
            })
            recv.capture(limit=None, timeout=None, wakeup=True)

    my_monitor()

There is no `runtime` attribute on a celery task. – Tristan Brown Sep 23 '19 at 22:28 — Tristan Brown, Sep 23 '19 at 22:28

Ben McAlindin · Answer 2 · 2015-12-31T20:12:53.473

When you create your tasks store a dict of {task_id: start_time}

I'm not sure how you're getting your results, but assuming you're doing something like:

for task in tasks:
    result = task.get()

You could change that to:

end_times = {}
While len(end_times) != len(tasks):
    for task in tasks:
        if task.ready():
            result = task.get()
            end_times[task.id] = now() #proper datetime method

And then work out your lengths from start and end

There may be a better way, but I'm just about to go out for new years.

Edit: You may need to handle tasks that have already been retrieved

How to measure Celery task duration

2 Answers2