How can I work around Argo output parameter size limit?

Question

I have an Argo workflow that loops over a JSON array. When the list gets too large, I get an error like this:

time="some-time" level=fatal msg="Pod \"some-pod-name\" is invalid: metadata.annotations: Too long: must have at most 262144 characters"

Or, in newer versions of Argo:

Output is larger than the maximum allowed size of 256 kB, only the last 256 kB were saved

How can I loop over this large JSON array without hitting the size limit?

My workflow looks a bit like this, but with a bigger JSON array:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: item
            value: "{{item}}"
        withParam: "{{steps.get-items.outputs.parameters.items}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items"]
    outputs:
      parameters:
      - name: items
        valueFrom:
          path: /tmp/items
  - name: echo
    inputs:
      parameters:
      - name: item
    container:
      image: stedolan/jq:latest
      command: [echo, "{{inputs.parameters.item}}"]

score 8 · Accepted Answer · answered Jun 23 '20 at 14:28

Instead of writing the JSON array to an output parameter, write it to an artifact and its length to an output parameter. Then you can use withSequence to loop over the array indexes and retrieve the corresponding item from the JSON artifact.

Here's an example with a hard-coded JSON array and item count. Your get-items step will certainly be more sophisticated.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: index
            value: "{{item}}"
          artifacts:
          - name: items
            from: "{{steps.get-items.outputs.artifacts.items}}"
        withSequence:
          count: "{{steps.get-items.outputs.parameters.count}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items && echo '3' > /tmp/count"]
    outputs:
      artifacts:
      - name: items
        path: /tmp/items
      parameters:
      - name: count
        valueFrom:
          path: /tmp/count
  - name: echo
    inputs:
      parameters:
      - name: index
      artifacts:
      - name: items
        path: /tmp/items
    container:
      image: stedolan/jq:latest
      command: [sh, -c]
      args: ["cat /tmp/items | jq '.[{{inputs.parameters.index}}]'"]

How can I work around Argo output parameter size limit?

1 Answers1

Linked