Specifying retry limit for tasks queued using GAE deferred library

Question

We are offloading certain time consuming tasks using the GAE deferred library and would like to know how can we set the retry limit for those offloaded tasks. We are running into issues where certain tasks are retried for ever as the task would never succeed due to some un recoverable exception.

Possible duplicate of [Retry count in deferred.defer in GAE](http://stackoverflow.com/questions/19579606/retry-count-in-deferred-defer-in-gae) — Dan Cornilescu, Apr 13 '16 at 17:34
Link you have provided talks about controlling the retry at application by throwing exception. I am wondering whether it is possible to specify the retry count while posting the job and deferred library takes care of enforcing it. — keerthy, Apr 14 '16 at 05:48

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

According to the documentation the _retry_options of the deferred.defer API can be used to pass retry options to the associated Task() instance:

_countdown, _eta, _headers, _name, _target, _transactional, _url, _retry_options, _queue: Passed through to the task queue - see the task queue documentation for details.

From the Task() doc:

...

And you can use the TaskRetryOptions()'s task_retry_limit property:

task_retry_limit

The maximum number of retry attempts for a failed task.

In push queues, the counter is incremented each time App Engine tries the task, up to the specified task_retry_limit. If specified with task_age_limit, App Engine retries the task until both limits are reached.

In pull queues, the counter is incremented each time the task is leased, up to the specified task_retry_limit. Tasks are deleted automatically once they have been leased the number of times specified in the limit.

Note: the answer is based on documentation only, I didn't actually implemented it, YMMV.

Thanks a lot for the response Dan. After going over the deferred library code, I figured out that _retry_options can be passed to get what I was looking for. I tried it as well and its working. — keerthy, Apr 14 '16 at 16:23

score 1 · Answer 2 · answered Apr 14 '16 at 09:04

According to the documentation

queue:
- name: fooqueue
  rate: 1/s
  retry_parameters:
    task_retry_limit: 7
    task_age_limit: 2d
- name: barqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 0
- name: bazqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 3

score 0 · Answer 3 · answered Apr 14 '16 at 09:20

Check X-Appengine-Taskretrycount and X-Appengine-Taskexecutioncount http header values in your task.

If you don't want to retry a task, you can raise deferred.PermanentTaskFailure exception. This exception will be logged only, task won't run again.

Different ways to access http headers:

For http handlers triggered by taskqueue: num_tries = self.request.headers.get('X-AppEngine-TaskRetryCount')

For functions triggered by deferred library: num_tries = webapp2.get_request().headers.get('X-AppEngine-TaskRetryCount')

Specifying retry limit for tasks queued using GAE deferred library

3 Answers3

Linked