0

I am using push task queue from GAE (python). There are times when after X minutes, Y% of the tasks have failed. For this situation, I want to purge the task queue (there is no need to execute them, eventually many will fail).

I can configure for a task to stop executing if it retries more than 2 times, but if I have 100 tasks that failed (300 runs = 100 + 200 retries) how can I stop the remaining tasks to execute?

queue.yaml:

queue:
- name: my-queue
  mode: push
  rate: 1/s
  bucket_size: 10
  max_concurrent_requests: 10
  retry_parameters:
    task_retry_limit: 2
octavian1001
  • 314
  • 2
  • 7

1 Answers1

1

I would store some values in memcache, like number of tasks in queue and timestamps of tasks that failed.

Each task would need to perform these tasks:

  • on start, calculate the rolling percentage (within last X minutes) and exit if the percentage is too high, or increment the number of tasks counter and proceed.
  • on failure, decrement the number of tasks counter and add a timestamp to the failed tasks list.
  • on success, decrement the number of tasks counter.

To calculate the rolling percentage, take the entire list of failed tasks and filter out those timestamps that are too old (over X minutes ago). Put the new list back into memcache. Then, take the number of tasks counter and calculate 100.0 * (number of failed tasks) / (number of tasks) to get your percentage. If it exceeds the Y% threshold, exit your task immediately.

Brent Washburne
  • 12,904
  • 4
  • 60
  • 82