How to export metrics from a containerized component in kubeflow pipelines 0.2.5

Question

I have a pipeline made up out of 3 containerized components. In the last component I write the metrics I want to a file named /mlpipeline-metrics.json, just like it's explained here. This is the Python code I used.

metrics = {
    'metrics': [
        {
            'name': 'accuracy',
            'numberValue':  accuracy,
            'format': 'PERCENTAGE',
        },
        {
            'name': 'average-f1-score',
            'numberValue': average_f1_score,
            'format': 'PERCENTAGE'
        },
    ]
}

with open('/mlpipeline-metrics.json', 'w') as f:
    json.dump(metrics, f)

I also tried writing the file with the following code, just like in the example linked above.

with file_io.FileIO('/mlpipeline-metrics.json', 'w') as f:
    json.dump(metrics, f)

The pipeline runs just fine without any errors. But it won't show the metrics in the front-end UI.

I'm thinking it has something to do with the following codeblock.

def metric_op(accuracy, f1_scores):
    return dsl.ContainerOp(
        name='visualize_metrics',
        image='gcr.io/mgcp-1190085-asml-lpd-dev/kfp/jonas/container_tests/image_metric_comp',
        arguments=[
            '--accuracy', accuracy,
            '--f1_scores', f1_scores,
        ]
    )

This is the code I use to create a ContainerOp from the containerized component. Notice I have not specified any file_outputs. In other ContainerOp I have to specify file_outputs to be able to pass variables to the next steps in the pipeline. Should I do something similar here to map the /mlpipeline-metrics.json onto something so that kubeflow pipelines detects it?

I'm using a managed AI platform pipelines deployment running Kubeflow Pipelines 0.2.5 with Python 3.6.8.

Any help is appreciated.

Can you please share your code? Does the file mlpipeline-metrics.json gets generated within the image? — Sourabh Jain, Apr 29 '20 at 12:52
The working code is as follows: Define the dictionary metrics and generate the mlpipeline-metrics.json inside the image. I used `with open('/mlpipeline-metrics.json', 'w') as f:` to create the file. When creating your `ContainerOp`, as stated below in the answer, you have to define the file_outputs. Make sure you use the exact same naming as I did and you should be good. — Jonas Boecquaert, May 04 '20 at 07:37

score 2 · Accepted Answer · answered Mar 18 '20 at 15:01

So after some trial and error I finally came to a solution. And I'm happy to say that my intuition was right. It did have something to do with the file_outputs I didn't specify. To be able to export your metrics you will have to set file_outputs as follows.

def metric_op(accuracy, f1_scores):
    return dsl.ContainerOp(
        name='visualize_metrics',
        image='gcr.io/mgcp-1190085-asml-lpd-dev/kfp/jonas/container_tests/image_metric_comp',
        arguments=[
            '--accuracy', accuracy,
            '--f1_scores', f1_scores,
        ],
        file_outputs={
            'mlpipeline-metrics': '/mlpipeline-metrics.json'
        }
    )

score 0 · Answer 2 · answered Jan 12 '21 at 09:51

Here is another way of showing metrics when you write python functions based method:

# Define your components code as standalone python functions:======================
    def add(a: float, b: float) -> NamedTuple(
        'AddOutput',
        [
            ('sum', float),
            ('mlpipeline_metrics', 'Metrics')
        ]
    ):
        '''Calculates sum of two arguments'''
        sum = a+b

        metrics = {
            'add_metrics': [
                {
                    'name': 'sum',
                    'numberValue': float(sum),
                }
            ]
        }
        print("Add Result: ", sum) # this will print it online in the 'main-logs' of each task


        from collections import namedtuple
        addOutput = namedtuple(
            'AddOutput',
        ['sum', 'mlpipeline_metrics'])
        return addOutput(sum, metrics)  # the metrics will be uploaded to the cloud

Note: I am jsut using a basci function here. I am not using your function.

How to export metrics from a containerized component in kubeflow pipelines 0.2.5

2 Answers2