2

I'd like to use a custom list to run parallel Ops in a Kubeflow Pipeline, and I want to use the value of the element of the list into the definition of the Op. I'm trying something like this:

my_list = ['foo', 'bar']
with dsl.ParallelFor(my_list) as item:
    op_first = dsl.ContainerOp(
        name=f'{item} - First Op',
        image=f'gcr.io/...',
        arguments=[
            ...
        ]
    )
    ...

But I'm getting an error like this:

ValueError: Only letters, numbers, spaces, "_", and "-"  are allowed in name.
Must begin with letter: {{pipelineparam:op=;name=loop-item-param-103a50f1}} - First Op

I've also tried

my_dict = [{'name': 'foo'}, {'name': 'bar'}]
with dsl.ParallelFor(my_list) as item:
    name = item.name
    op_first = dsl.ContainerOp(
        name=f'{name} - First Op',
        image=f'gcr.io/...',
        arguments=[
            ...
        ]
    )
    ...

But i get a similar error. How can I retrieve the "original" name of the item?

Matteo Felici
  • 1,037
  • 10
  • 19

2 Answers2

4

I know this is an old question, but I just encountered the same issue, so hopefully, this will help somebody else.

For the second option you tried, replace the dictionary key 'name' with something else as below.

my_dict = [{'job_name': 'foo'}, {'job_name': 'bar'}]

It seems that the key 'name' is used internally by Kubeflow, and assigning a value to it causes your error.

Andrew Chalk
  • 41
  • 2
0

I think the problem is the name you are giving to the ContainerOp.

name=f'{name} - First Op'

Just name=f'{name} should work

jspablo
  • 1