failed to save outputs: timed out waiting for the condition on Argo Workflow

Question

I'm running Argo workflow on Kubernetes. And after I followed this blog to setup Jupyterhub, I started getting this error (never had the issue before using Jupyterhub) on the pods on Argo: failed to save outputs: timed out waiting for the condition.

The job always fails if I add

  resources:
    limits:
      nvidia.com/gpu: 1

But if gpu is not used, it sometimes succeeds (with retryStrategy after occasional failures).

Could someone help me out?

could you paste logs of workflow controller? `kubectl logs -n argo workflow-controller-XXXX` — weitang114, Jul 30 '19 at 03:37
@user3368526 when you add `resources: limits: nvidia.com/gpu: 1` it fails with the same error? did you try to increase `activeDeadlineSeconds` parameter for argo? — A_Suh, Jul 30 '19 at 09:50
@A_Suh Thanks, yes it fails with that same error. I increased the activeDeadlineSeconds but it didn't change anything. It seems like the first step is not able to access or save the artifact in /tmp/ directory to share the artifacts across the steps. — user3368526, Jul 31 '19 at 05:30
can you list your pods? `kubectl get pods --all-namespaces o wide` — A_Suh, Aug 22 '19 at 12:57
Similar error message I am also getting with Argo 2.10.1 "failed to save outputs: Failed to establish pod watch: timed out waiting for the condition" — Black_Rider, Sep 04 '20 at 16:03

score 0 · Answer 1 · answered Sep 01 '22 at 07:41

0

check if you have mentioned the serviceAccountName

workflow.spec.serviceAccountName

for more info check this link

answered Sep 01 '22 at 07:41

Mukul Gaur

1

failed to save outputs: timed out waiting for the condition on Argo Workflow

1 Answers1