I have a large list-output and would like to run a separate task for each entry. For small outputs I can use below template. But If I increase output, e.g.
for run in {1..100000}; do
it crashes. How can I solve this problem? I tried to use artifacts, but for artifacts I cannot iterate elements of a list.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: example-large-output-
spec:
entrypoint: main
templates:
- name: main
steps:
- - name: create-large-output
template: create-large-output-template
- - name: iterate-large-output
template: iterate-large-output-template
arguments:
parameters:
- name: fp
value: "{{item}}"
withParam: "{{steps.create-large-output.outputs.result}}"
- name: create-large-output-template
script:
image: debian:9.4
command: [bash]
source: |
o='['
for run in {1..100}; do
o=$o'"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",'
done
o=${o::-1}"]"
echo $o
- name: iterate-large-output-template
inputs:
parameters:
- name: fp
script:
image: alpine
command:
- sh
source: |
echo {{inputs.parameters.fp}}