Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
3
votes
2 answers

Kubeflow Pipeline RuntimeError while running multiple trainers with tfx

I like to have multiple trainers running simultaneously using the same ExampleGen, Schema and Transform. Below is my code adding extra components as trainer2 evaluator2 and pusher2. But I've been getting the following error, and I'm not sure how to…
LLTeng
  • 385
  • 3
  • 4
  • 15
3
votes
1 answer

Upload pipeline on kubeflow

I am currently trying to setup a kubeflow pipeline. My use case requires that the configuration for pipelines shall be provided via a yaml/json structure. Looking into the documentation for submitting pipelines I came across this paragraph: Each…
LexByte
  • 382
  • 4
  • 15
3
votes
2 answers

How to pass an environmental variable in kubeflow pipeline?

I want the variable to be accessed by gcr.io/******/serve_model:lat5 Image which is an argument of gcr.io/******/deployservice:lat2 Initially I have tried passing the variable as argument but it didn't work, so I am trying to pass it as an…
harish kumaar
  • 41
  • 1
  • 3
3
votes
2 answers

In Kubeflow Pipelines, how to send a list of elements to a lightweight python component?

I am trying to send a list of elements as a PipelineParameter to a lightweight component. Here is a sample that reproduces the problem. Here is the function: def my_func(my_list: list) -> bool: print(f'my_list is {my_list}') …
Kevin Pauli
  • 8,577
  • 15
  • 49
  • 70
2
votes
1 answer

Vertex AI Pipelines. Batch Prediction 'Error state: 5.'

I have been trying to run a Vertex AI pipeline using Kubeflow Pipelines and the google-cloud-pipeline-components library. The pipeline is entirely custom container components with the exception of the batch predictions. The code for my pipeline is…
AndrewJaeyoung
  • 368
  • 3
  • 10
2
votes
1 answer

why is kubeflow on vertex ai pipelines not storing metadata for dataset artifact

I am trying to pass metadata between python function components by attaching it an output artifact in a vertex ai kubeflow pipeline, from the documentation this seems straightforwards, but try as I might I can't get it to work. I am trying to attach…
Gavin
  • 71
  • 1
  • 5
2
votes
0 answers

How to deploy Kubeflow Components with cloud build?

I am using Containerized Python components to create the components in my pipeline which works well but I cannot deploy them using Google Cloud Platform Cloud Build. steps: - name: gcr.io/cloud-builders/docker args: [ 'build', '-t', 'dcgp',…
Alexander Townsend
  • 249
  • 2
  • 10
2
votes
1 answer

Mounting an existing volume from a Kubeflow Pipeline: kfp.VolumeOP creates a new volume instead of creating a PVC to existing volume

I am new to KubeFlow and trying to port / adapt an existing solution to run in KubeFlow pipelines. The issue I am solving now is that the existing solution shared data via a mounted volume. I know this is not the best practice for components…
user3197263
  • 71
  • 3
2
votes
2 answers

Return parameters in Kubeflow pipeline component

Im trying to return several parameters from a kubeflow component. Following the documentation it says you should use OutputPath where T is one of the types in python (str, float, int...) so I have this: @component def…
Sergiodiaz53
  • 1,268
  • 2
  • 14
  • 23
2
votes
3 answers

How to test kfp components with pytest

I'm trying to local test a kubeflow component from kfp.v2.ds1 (which works on a pipeline) using pytest, but struggling with the input/output arguments together with fixtures. Here is a code example to illustrate the issue: First, I created a fixture…
Gabriel Caldas
  • 371
  • 2
  • 13
2
votes
1 answer

How to render a confusion matrix visualization in a Kubeflow pipeline component step?

Here is what I am trying from one of the official tutorials import kfp import kfp.dsl as dsl from kfp.components import create_component_from_func @create_component_from_func def confusion_visualization(matrix_uri: str =…
Diablo3093
  • 963
  • 4
  • 15
  • 26
2
votes
1 answer

Set the name for each ParallelFor iteration in KFP v2 on Vertex AI

I am currently using kfp.dsl.ParallelFor to train 300 models. It looks something like this: ... models_to_train_op = get_models() with dsl.ParallelFor(models_to_train_op.outputs["data"], parallelism=100) as item: prepare_data_op =…
Optimus
  • 1,354
  • 1
  • 21
  • 40
2
votes
0 answers

Kubeflow Pipelines v2 is giving Permission Denied on OutputPath

In Kubeflow Pipelines v2, running on EKS with default install, I'm getting a "permission denied" error. It ran correctly in KFP v1. time="2022-04-26T21:53:30.710Z" level=info msg="capturing logs" argo=true I0426 21:53:30.745547 18…
aronchick
  • 6,786
  • 9
  • 48
  • 75
2
votes
1 answer

Python OOP in Kubeflow Pipelines

I have a complete preprocessing, training, evaluating ML pipeline in Python 3.8 that I want to transform in a Kubeflow pipeline. It's mainly for segmentation task (i.e. on RGB images) All my code is design in a OOP way. Is this possible to keep the…
mabrtn
  • 21
  • 1
2
votes
0 answers

Is it possible to get current experiment name in kubeflow pipeline?

When in pipeline code I can use macros like {{workflow.annotations.pipelines.kubeflow.org/run_name}} or {{workflow.name}} to get those values, and potentially use them in some context, eg. passing to some component. What I'd like to do is get…
Marcin Cylke
  • 2,100
  • 2
  • 21
  • 40
1 2
3
…
18 19