How to get notified of Google App Engine Task Queue failure

Question

I am using a GAE push Task Queue. I want the task to run once and if it fails I want it to notify me somehow. I also don't want it to do any retries. If it fails I want it to sit in the queue until I rerun it manually.

I can manually catch all exceptions and send an email to myself it it fails. Is there a better way? In the queue config I can specify retry limit (see below) of zero but if I do this the task is deleted as soon as it fails.

# queue.yaml
- name: default
  rate: 5/s
  retry_parameters:
    task_retry_limit: 0

Any ideas?

score 5 · Answer 1 · answered Oct 03 '14 at 18:29

When a task retries, you can read the retry count in next retry. And do what you need to do.

Look here for : X-AppEngine-TaskRetryCount, the number of times this task has been retried; for the first attempt, this value is 0. This number includes attempts where the task failed due to a lack of available instances and never reached the execution phase.

score 3 · Accepted Answer · answered Oct 03 '14 at 20:08

The only way I can think of to accomplish this would be to set a high value for min_backoff_seconds. If you can set that for something high like 7 days, then that should give you enough time to take action in response to your email message.

I don't know if there is a limit to how high a value you can set for min_backoff_seconds so you'll want to check that.

I think catching exceptions and sending yourself an email is the only way to notify yourself of a task failure.

score 0 · Answer 3 · answered Oct 03 '14 at 16:07

I don't think of any way to do what you want besides grabbing all exceptions. GAE won't send a message when a taskqueue fails, there is no service to "allow" to get that functionality unfortunately.

Unless you get a Google Engineer to answer something about a service I'm not aware of....

guneysus · Answer 4 · 2014-11-28T12:53:10.690

0

@voscausa's answer means that: you can implement a taskCheckerTask that always returns http codes any number except between 200-299. But do not forget task will be deleted from queue if it fails.

class TaskChecker(webapp2.RequestHandler):
    def post(self):
        # Check tasks and retries here
        # If need send mail
        self.echo(500)

edited Nov 28 '14 at 12:53

answered Nov 28 '14 at 12:40

guneysus

6,203
2
45
47

How to get notified of Google App Engine Task Queue failure

4 Answers4