2

One of our Argo Workflow steps may hit a rate limit and I want to be able to tell argo how long it should wait until the next retry.

Is there a way to do it?

I've seen Retries on the documentation but it only talks about retry count and backoff strategies and it doesn't look like it could be parameterized.

Mithir
  • 2,355
  • 2
  • 25
  • 37

1 Answers1

2

As far as I know there's no built-in way to add a pause before the next retry.

However, you could build your own with Argo's exit handler feature.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: exit-handler-with-pause-
spec:
  arguments:
    parameters
    - name: pause-before-retry-seconds
      value: "60"
  entrypoint: intentional-fail
  onExit: exit-handler
  - name: intentional-fail
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo intentional failure; exit 1"]
  - name: exit-handler
    steps:
    - - name: pause
        template: pause
        when: "{{workflow.status}} != Succeeded"
  - name: pause
    container:
      image: alpine:latest
      env:
      - name: SECONDS
        value: "{{workflow.parameters.pause-before-retry-seconds}}"
      command: [sh, -c]
      args:
      - >-
        echo "Pausing before retry..."
        sleep "$SECONDS"

If the retry pause needs to be calculated within the workflow, check out the exit handler with params example.

crenshaw-dev
  • 7,504
  • 3
  • 45
  • 81
  • That's a nice approach, but we are using an active pod to wait, and this could potentially be long, but given that this is the simplest of scripts, can I assume minimal cpu/mem usage? – Mithir Jan 13 '22 at 06:33
  • @Mithir good question... I'm not really sure what the costs of the wait pod would be or how to estimate them. But yes, I'd think they should be relatively small. If the costs are too high, you could probably hack together a custom retry mechanism with exit handlers and a `resource` template creating a CronWorkflow. – crenshaw-dev Jan 13 '22 at 14:17
  • Is there a wait to activate a templateRef on hooks.exit? – Mithir Jan 13 '22 at 14:36
  • @Mithir I've never tried it, but I don't see any reason it wouldn't work. You'd just replace the `template: whatever` with a `templateRef` block. – crenshaw-dev Jan 13 '22 at 14:41
  • Even if `templateRef` didn't work nested underneath `hooks.exit`, you could reference a steps template whose only step is a `templateRef`. – crenshaw-dev Jan 13 '22 at 14:43
  • For some reason I get "failed to get a template" when using templateRef under hooks.exit, it works the way you suggested it but it would have been a lot simpler with a templateRef – Mithir Jan 13 '22 at 19:10
  • 1
    @Mithir Could you submit an issue? I _think_ hook.exit is a relatively new feature, so you might be the first to have encountered the problem. – crenshaw-dev Jan 13 '22 at 19:11
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/241042/discussion-between-mithir-and-crenshaw-dev). – Mithir Jan 13 '22 at 19:14