We are using kpf v2 in Google Clouds Vertex AI and want to log metrics from a component built by a yaml specification.
We can only make this work from a Python function based component were we can call the mlpipeline-metrics (of type Output[Metrics]) with log_metrics(..) directly.
The docs at https://www.kubeflow.org/docs/components/pipelines/sdk/pipelines-metrics/ says that you can write your metrics in json from inside the container to an OutputPath named MLPipeline Metrics. But this doesn't work in Vertex AI since the metr8cs does not show up in the console.
Our component is set up like this:
name: a component
outputs:
- {name: MLPipeline Metrics, type: Metrics}
implementation:
container:
image: eu.gcr.io/project/knark/base:latest
command: [python3, -m, module.foo,
--metrics, {outputPath: MLPipeline Metrics}
With the metrics file being written like this:
parser = ArgumentParser()
parser.add_argument('-m', '--metrics', default='/mlpipeline-metrics.json')
opts = parser.parse_args()
metrics = {'cluster-entropy': .5, 'inertia': 100.}
with open(metrics_path, 'w') as f:
json.dump({'metrics': [{'name': m, 'value': v, 'format': 'RAW'} for m, v in metrics.items()]}, f)
We also tried setting
fileOutputs:
mlpipeline-metrics: /mlpipeline-metrics.json
in the component which seems to be the way to do it earlier but this doesn't appear to work either.
Are there any details I'm missing in writing out the metrics or does this not work at all or is it unsupported in Vertex AI?
