appengine runs failed tasks twice even if task_retry_limit=0

Question

I'm seeing a buggy behaviour in taskqueue API. When a task fails, appengine always runs it once again, even if I tell it not to.

This is the relevant code:

NO_RETRY = TaskRetryOptions(task_retry_limit=0)


class EnqueueTaskDapau(webapp2.RequestHandler):
    def get(self):
        taskqueue.add(
            url='/task_dapau',
            queue_name='DEFAULT',
            retry_options=NO_RETRY
        )


class TaskDapau(webapp2.RequestHandler):
    def get(self):
        logging.warning('Vai dar pau')
        raise BaseException('Deu pau :-)')

    def post(self):
        return self.get()


application = webapp2.WSGIApplication([
    ('/', MainPage),
    ('/enqueue_dapau', EnqueueTaskDapau),
    ('/task_dapau', TaskDapau),
], debug=True)

The whole app is available on Github so it should be easy to reproduce. When I point my browser to /enqueue_dapau, this is what I see in the logs (on the web console):

2014-10-30 08:31:01.054 /task_dapau 500 4ms 0kb AppEngine-Google; (+http://code.google.com/appengine) module=default version=1
W 2014-10-30 08:31:01.052 Vai dar pau
E 2014-10-30 08:31:01.053 Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in

2014-10-30 08:31:00.933 /task_dapau 500 3ms 0kb AppEngine-Google; (+http://code.google.com/appengine) module=default version=1
W 2014-10-30 08:31:00.931 Vai dar pau
E 2014-10-30 08:31:00.932 Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in

2014-10-30 08:31:00.897 /enqueue_dapau 200 91ms 0kb Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36 module=default version=1

If I look at Task Queues on the web console, I see "Run in Last Minute == 2" This behaviour is different from what I get locally with the SDK:

INFO     2014-10-30 15:49:05,711 module.py:666] default: "GET /enqueue_dapau HTTP/1.1" 200 -
WARNING  2014-10-30 15:49:05,729 views.py:33] Vai dar pau
ERROR    2014-10-30 15:49:05,729 wsgi.py:279] 
Traceback (most recent call last):
  File "/home/tony/google_appengine/google/appengine/runtime/wsgi.py", line 267, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/home/tony/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/tony/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/tony/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/home/tony/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/home/tony/work/qmag/gaetests/src/views.py", line 37, in post
    return self.get()
  File "/home/tony/work/qmag/gaetests/src/views.py", line 34, in get
    raise BaseException('Deu pau :-)')
BaseException: Deu pau :-)
INFO     2014-10-30 15:49:05,735 module.py:666] default: "POST /task_dapau HTTP/1.1" 500 -
WARNING  2014-10-30 15:49:05,735 taskqueue_stub.py:1986] Task task4 failed to execute. The task has no remaining retries. Failing permanently after 0 retries and 0 seconds

Is this a bug? (It really looks like so)

Is there an easy workaround for it?

Filed a bug on their tracker - https://code.google.com/p/googleappengine/issues/detail?id=11425 — Tony Lâmpada, Oct 30 '14 at 17:10
Related: http://stackoverflow.com/questions/5505416/google-app-engine-task-retry-limit-doesnt-work — Tony Lâmpada, Oct 30 '14 at 18:15

score 3 · Answer 1 · answered Oct 30 '14 at 20:50

3

As mentioned in the documentation App Engine will sometimes run a task twice. You should write your tasks to ensure that this will not be harmful.

answered Oct 30 '14 at 20:50

David

9,288
1
20
52

Geeze thanks! They should mention this *very close* to where they say what 'task_retry_limit' was supposed to do >:| – Tony Lâmpada Oct 30 '14 at 21:06

score 3 · Answer 2 · answered Oct 31 '14 at 12:08

I just found a way to avoid the undesired retry:

taskqueue.add(
    url='/blah',
    queue_name='myq',
    retry_options=TaskRetryOptions(task_retry_limit=0, task_age_limit=1),
    countdown=1,
)

This combination of of retry_limit, age_limit and countdown is the magical incantation that does the trick.

It's still suboptimal though, so I'll leave this without a green answer until google fixes this bug.

score -1 · Answer 3 · answered Nov 04 '14 at 23:01

-1

Check your queue.yaml file and make sure it is correctly configured.

queue:
- name: default
  retry_parameters:
    task_retry_limit: 0

answered Nov 04 '14 at 23:01

maciekrb

227
1
11

appengine runs failed tasks twice even if task_retry_limit=0

3 Answers3

Linked