In Kubeflow Pipelines, how to send a list of elements to a lightweight python component?

Question

I am trying to send a list of elements as a PipelineParameter to a lightweight component.
Here is a sample that reproduces the problem. Here is the function:

def my_func(my_list: list) -> bool:
    print(f'my_list is {my_list}')
    print(f'my_list is of type {type(my_list)}')
    print(f'elem 0 is {my_list[0]}')
    print(f'elem 1 is {my_list[1]}')
    return True

And if I execute it with this:

test_data = ['abc', 'def']
my_func(test_data)

It behaves as expected:

my_list is ['abc', 'def']
my_list is of type <class 'list'>
elem 0 is abc
elem 1 is def

but if I wrap it in an op and and set up a pipeline:

import kfp

my_op = kfp.components.func_to_container_op(my_func)

@kfp.dsl.pipeline()
def my_pipeline(my_list: kfp.dsl.PipelineParam = kfp.dsl.PipelineParam('my_list', param_type=kfp.dsl.types.List())):
    my_op(my_list)

kfp.compiler.Compiler().compile(my_pipeline, 'my_pipeline.zip')

And then run a pipeline:

client = kfp.Client()
experiment = client.create_experiment('Default')
client.run_pipeline(experiment.id, 'my job', 'my_pipeline.zip', params={'my_list': test_data})

Then it seems at some point my list was converted to a string!

my_list is ['abc', 'def']
my_list is of type <class 'str'>
elem 0 is [
elem 1 is '

score 2 · Accepted Answer · answered Sep 05 '19 at 17:51

Here is a workaround I discovered, serializing arguments as a json string. Not sure this is really the best way...

The bare function becomes:

def my_func(json_arg_str: str) -> bool:
    import json
    args = json.loads(json_arg_str)
    my_list = args['my_list']
    print(f'my_list is {my_list}')
    print(f'my_list is of type {type(my_list)}')
    print(f'elem 0 is {my_list[0]}')
    print(f'elem 1 is {my_list[1]}')
    return True

Which still works as long as you pass the args as a json string instead of a list:

test_data = '{"my_list":["abc", "def"]}' my_func(test_data)

Which produces expected results:

my_list is ['abc', 'def']
my_list is of type <class 'list'>
elem 0 is abc
elem 1 is def

And now the pipeline is changed to accept a str instead of a PipelineParam of type kfp.dsl.types.List:

import kfp 

my_op = kfp.components.func_to_container_op(my_func)

@kfp.dsl.pipeline()
def my_pipeline(json_arg_str: str):
    my_op(json_arg_str)

kfp.compiler.Compiler().compile(my_pipeline, 'my_pipeline.zip')

Which, when executed like this:

client = kfp.Client()
experiment = client.create_experiment('Default')
client.run_pipeline(experiment.id, 'my job', 'my_pipeline.zip', params={'json_arg_str': test_data})

Produces the same result:

my_list is ['abc', 'def']
my_list is of type <class 'list'>
elem 0 is abc
elem 1 is def

Although it works, I nevertheless find this workaround annoying. What then is the point of kfp.dsl.types.List, if not for allowing a PipelineParam that is a List?

score 0 · Answer 2 · answered Sep 12 '19 at 17:52

0

Currently the best option seems to be serializing the arguments. There is one issue related to this: https://github.com/kubeflow/pipelines/issues/1901

answered Sep 12 '19 at 17:52

Gabriel Bessa

468
1
4
8

In Kubeflow Pipelines, how to send a list of elements to a lightweight python component?

2 Answers2