Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.
Questions tagged [kubeflow-pipelines]
273 questions
4
votes
1 answer
Constructing a Vertex AI Pipeline with a custom training container and a model serving container
I'd like to be able to train a model with a training app container that I've made and saved to my artifact registry. I want to be able to deploy a model with a flask app and with a /predict route that can handle some logic -- not necessarily just…
Sanger Steel
- 51
- 4
4
votes
3 answers
How to connect kubeflow pipeline components
I want to establish a pipeline connection between the components by passing any kind of data just to make it look like organized like flowchart with arrows. Right now it is like below
Irrespective of whether the docker container generates output or…
Sagar SN
- 35
- 1
- 8
4
votes
1 answer
How to skip already-run steps in kubeflow pipeline?
I'm building an ML pipeline in Kubeflow and I have a question. Is there anything out of the box that allows me to configure my pipeline, such that a step is not rerun if its output exists? I've thought of ways to do this manually (either checking…
Elisabeth Guegan
- 109
- 1
- 5
3
votes
0 answers
Access information from a previous Kubeflow component
I have a ModelBatchPredictOp component in my pipeline. This component generates 3 artifacts: batchpredictionjob, big_query_table, and gcs_output_directory. The pipeline is running fine.
What I need is a way to access the tableId property of artifact…
Openworld
- 31
- 3
3
votes
2 answers
GCP Vertex AI Endpoint returning empty prediction array
KFP pipeline job executes successfully, but upon hitting the endpoint, am getting an empty predictions array ([]). I suspect the issue is in the model upload, where the model is not registered correctly somehow. Any tips are appreciated.
Code to…
juldavis
- 31
- 2
3
votes
1 answer
kubeflow component - why so many ways to define a component and what are the differences?
Please help understand what are the meaningful/significant differences among different ways to create kubeflow pipeline components and the reason for having so many ways?
from kfp.components import func_to_container_op
@func_to_container_op
def…
mon
- 18,789
- 22
- 112
- 205
3
votes
0 answers
KUBEFLOW: no such file or directory Error
I am creating a very simple kubeflow pipeline which has the following steps:
Data Loading
Data Preprocessing
Training
Evaluation
The first 3 steps runs successfully but as soon as I try to run the evaluator the step fails with the error mentioned…
R.Ahuja
- 51
- 4
3
votes
1 answer
How can I programmatically create a Kubeflow recurring run from a pipeline function?
I am trying to create a recurring kubeflow pipeline run as follows:
from kfp import compiler
compiler.Compiler().compile(
pipeline_func=my_pipeline,
package_path='pipelines/my_pipeline.tgz')
from kfp.v2.google.client import…
jtlz2
- 7,700
- 9
- 64
- 114
3
votes
1 answer
Autoscaling VertexAI pipeline components
I am exploring VertexAI pipelines and understand that it is a managed alternative to, say, AI Platform pipelines (where you have to deploy a GKE cluster to be able to run Kubeflow pipelines). What I am not clear on is whether VertexAI will autoscale…
racerX
- 930
- 9
- 25
3
votes
1 answer
Skip failed steps in kubeflow pipeline and continue the pipeline
Does anyone have an example how to handle failed steps in pipeline and continue pipeline even after failed step. ExitHandler can execute only the one, last step while I need to catch any possible failure i.e. something more like (pseudocode)
step1 =…
fab
- 31
- 2
3
votes
1 answer
What is the difference between volume and pvolume in kubeflow pipeline definition?
I am studying kubeflow pipelines and how are the different components of the pipeline linked to each other. For this, I am using an example of MNIST project available on the official GitHub repository. But I am not able to understand the difference…
shrikanth singh
- 95
- 8
3
votes
1 answer
Get experiment name from within a Kubeflow pipeline run
I just started working with Kubeflow and I ran into a problem. I need my pipeline to be able to automatically get the name of the experiment it belongs to. I tried to use the kfp package but it seems to me that there is no way to get the experiment…
user2846
- 133
- 2
3
votes
2 answers
Connecting to AI Platform Pipelines using the Kubeflow Pipelines SDK with Python and PyCharm
Using windows 10 and Python 3.9 with PyCharm IDE
I am trying to list the pipelines using the following code:
import kfp
client = kfp.Client(host='.......bb92ea05-dot-us-central2.pipelines.googleusercontent.com')
client.list_pipelines()
I know…
Franco
- 441
- 3
- 18
3
votes
0 answers
Google Cloud Platform Kubeflow Pipeline Error
I have a Google Cloud Platform account with a Kubeflow Pipeline. The first component of the pipeline preprocesses some data and the second one trains a model (SKlearn Decision Tree Classifier) with that preprocessed data. For the purpose of showing…
Amorim95
- 51
- 4
3
votes
1 answer
Allocate resources for Kubeflow pipeline using pipeline params
I would like to be able to create a Kubeflow pipeline that allows users to set the allocated resources for a run. The end result would be something like this:
Example of Kubeflow "Create Run" UI with ability to set resource allocation.
Definition of…
Carter
- 33
- 3