Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.
Questions tagged [kubeflow-pipelines]
273 questions
2
votes
1 answer
Kubeflow-kale :- How to integrate kubeflow-kale extension to run pipelines on a seperate standalone cluster of Kubeflow pipelines
I am currently trying to use the kubeflow kale jupyter extension on my local jupyterlab server without Kubernetes and kubeflow installed and trying to run my code pipeline on GCP AI pipeline server or any other Cloud Kubeflow pipeline server. I am…
Rituraj kumar
- 349
- 1
- 4
- 15
2
votes
1 answer
How to authenticate with Kubeflow Pipelines API
I'm hoping to experiment with triggering pipelines using the raw Kubeflow Pipeline API, but there is very little information about how to authenticate. All the information I can find in…
Jonas D
- 258
- 5
- 14
2
votes
1 answer
Kubeflow without Google Cloud Storage
Is it possible to replace the usage of Google Cloud Storage buckets with an alternative on-premises solution so that it is possible to run e.g. Kubeflow Pipelines completely independent from the Google Cloud Platform?
2
votes
2 answers
How can you specify local path of InputPath or OutputPath in Kubeflow Pipelines
I've started using Kubeflow Pipelines to run data processing, training and predicting for a machine learning project, and I'm using InputPath and OutputhPath to pass large files between components.
I'd like to know how, if it's possible, do I set…
Jonas D
- 258
- 5
- 14
2
votes
2 answers
How to export metrics from a containerized component in kubeflow pipelines 0.2.5
I have a pipeline made up out of 3 containerized components. In the last component I write the metrics I want to a file named /mlpipeline-metrics.json, just like it's explained here.
This is the Python code I used.
metrics = {
'metrics': [
…
Jonas Boecquaert
- 51
- 6
2
votes
2 answers
Is it possible to let a Kubeflow inference pipeline (KFServing) publish messages to Kafka?
I'm beginning to use Kubeflow and I've read about KFServing. There is this integration with Kafka to consume from a topic. What I need is to run an inference pipeline that consumes data from Kafka, does some data preproccessing, run the ML…
openrory
- 61
- 6
2
votes
2 answers
How to build an image for KubeFlow pipeline?
I recently found out about kubeflow and kubeflow pipeline but it is not clear for me how to build an image from my python program.
Let's assume that I have a simple python function that crops images:
class Image_Proc:
def crop_image(self, image,…
AVarf
- 4,481
- 9
- 47
- 74
2
votes
1 answer
kubeflow ParallelFor using the previous containerop output
I can create a static for loop using
with dsl.ParallelFor([1,2,3]) as item:
....
How can I use a container_op.output as an input to ParallelFor?
Assume the first container outputs an integer n, and then I want to run ParallelFor n…
user3599803
- 6,435
- 17
- 69
- 130
1
vote
2 answers
AI Platform Pipeline 2.0.0 deployment not visible on GCP Console
On GCP, I created a new kubernetes cluster 1.26 and deployed kubeflow pipelines 2.0.0 on it. The deployment went fine but on the AI Platform Pipeline page, I don't see the new deployment. Due to this, I am unable to open the newly deployed 2.0.0 kfp…
Behroz Sikander
- 3,885
- 3
- 22
- 36
1
vote
1 answer
Adding arbitrary metadata to kubeflow pipeline component
Kubeflow pipelines docs mentions that there is a way to add metadata annotations as a key-value pairs. From the other hand, I could not find any relevant documentation of how it can be done.
Is there an option to add some metadata to any kfp…
Omri
- 43
- 1
- 4
1
vote
0 answers
how to get model from KF Containerized Python Component into Vertex AI model registry properly
if custom model training happens in Containerized Python Component, producing model file and metrics, what is the proper way of uploading the model and its metrics into Vertex AI so that they are available via Vertex AI UI?
Google has changed almost…
Maxim Volgin
- 3,957
- 1
- 23
- 38
1
vote
0 answers
what replaces `aiplatform.CustomContainerTrainingJobRunOp` in `google-cloud-pipeline-components` V2?
Apparently this class I've been using heavily is removed in V2 and it is not exactly clear from the docs what it should be replaced with.
"Remove aiplatform.CustomContainerTrainingJobRunOp and aiplatform.CustomPythonPackageTrainingJobRunOp…
Maxim Volgin
- 3,957
- 1
- 23
- 38
1
vote
0 answers
How to trigger a component in a Kubeflow pipeline regardless of whether a previous condition is skiped or completed
My graph starts with a common component after which, the process is split in two leaves, one for variables and the second for events. both leaves usea a conditional to execute some components if there is data available.
After procesing both leaves…
1
vote
1 answer
Vertex AI pipeline run fails after passing the dsl.Condition==True
I have been working with Vertex AI pipelines for 3 months. I started with migrating projects from old dedicated Kubernetes infrastructure to Vertex AI Python component based pipelines. I created a template pipelines which were running fine until…
Pawan Kumar
- 31
- 2
1
vote
1 answer
How to fix TypeError: to_dict() missing 1 required positional argument: 'self' on a dsl.component
I am creating a pipeline on a VertexAI Workbench using Kubeflow Lightweight components.
I start with a component to extract data from BigQuery and process it (and returning a dataframe).
This dataframe will then be fed to another…
filipe
- 275
- 1
- 1
- 8