Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.
Questions tagged [kubeflow-pipelines]
273 questions
0
votes
1 answer
Using InputPath/Output Path on Vertex?
TLDR, does Google Cloud Vertex support InputPath/OutputPath?
I have set up my environment on Google Cloud in the following way:
export PROJECT_ID="xxx"
export SERVICE_ACCOUNT_ID="xxx"
export USER_EMAIL="xxx"
export BUCKET_NAME="xxx"
export…
aronchick
- 6,786
- 9
- 48
- 75
0
votes
1 answer
kubeflow - how to connect to kubeflow pipeline endpoint from the Jupyter notebook in the kubeflow
Problem
From within the kubeflow jupyter notebook, the connection to the kubeflow pipeline fails, although followed the Connect to Kubeflow Pipelines from the same cluster - Multi-user mode.
import os
import kfp
with…
mon
- 18,789
- 22
- 112
- 205
0
votes
1 answer
Go-cron expression to trigger on 3rd Thursday of every month
How to write a go-cron expression to trigger a run of a job on the 3rd Thursday of every month? This is specific to scheduling pipelines on Kubeflow.
For example: croniter library provides the hash symbol for day of the week which allows you to…
0
votes
1 answer
How to get the status of a pipeline run within a component, running on Vertex AI?
Previously, using Kubeflow Pipelines SDK v1, the status of a pipeline could be inferred during pipeline execution by passing an Argo placeholder, {{workflow.status}}, to the component, as shown below:
import kfp.dsl as dsl
component_1 =…
Marcus
- 943
- 5
- 21
0
votes
1 answer
GCP Vertex Pipeline - Why kfp.v2.dsl.Output as function arguments work without being provided?
Why kfp.v2.dsl.Output as function argument works without being provided?
I am following Create and run ML pipelines with Vertex Pipelines! Jupyter notebook example from GCP.
The function classif_model_eval_metrics takes metrics: Output[Metrics] and…
mon
- 18,789
- 22
- 112
- 205
0
votes
1 answer
Kubeflow Pipelines error on GCP - Run doesn't end
After submitting the "run" using Jupyter notebook, when I go to the Kubeflow pipeline dashboard, I can see my "run" submitted & running but it doesn't end even after few hours & doesn't show any logs either.
I have tried running the "DEMO XG-BOOST"…
sai r
- 1
- 2
0
votes
0 answers
Vertex AI Pipelines: APIs from v1 namespace not supported in v2 compiler
I started taking my first steps in Vertex AI on past week. As you can see, that basic code works, however it displays the following warning:
/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py:1266: FutureWarning: APIs…
David Espinosa
- 760
- 7
- 21
0
votes
1 answer
Is there an implemented way to use a kubeflow pipeline's output outside the pipeline?
I'm using local kubeflow pipelines for building a continuous machine learning test project. I have one pipeline that preprocess the data using TFX, and it saves the outputs automatically to minio. Outside of this pipeline, I want to train the model…
AnoderPersona
- 3
- 3
0
votes
1 answer
Kubeflow Pipelines How can we create static HTML visualization with embedded iFrame using inline storage?
I am wondering how could I create a simple static HTML visualization for kubeflow pipelines using inline storage?
My use case is I'd like to pass a string with raw html containing a simple iframe.
The sample from the doc does not work for me (kfp…
Alexandre Brown
- 51
- 5
0
votes
1 answer
ModelUploadOp step failing with custom prediction container
I am currenlty trying to deploy a Vertex pipeline to achieve the following:
Train a custom model (from a custom training python package) and dump model artifacts (trained model and data preprocessor that will be sed at prediction time). This is…
anddt
- 1,589
- 1
- 9
- 26
0
votes
2 answers
kubeflow OutputPath/InputPath question when writing/reading multiple files
I've a data-fetch stage where I get multiple DFs and serialize those. I'm currently treating OutputPath as directory - create it if it doesn't exist etc. and then serialize all the DFs in that path with different names for each DF.
In a subsequent…
soumeng78
- 600
- 7
- 12
0
votes
0 answers
kubeflow pipeline Failed to execute component: unable to get pipeline with PipelineName
Install follow https://github.com/kubeflow/manifests in v1.4.1
KFP version: 1.7.0
KFP SDK version: build version dev_local
k3s Kubernetes 1.19
use demo example to add pipline
kfp 1.8.10
kfp-pipeline-spec 0.1.13
kfp-server-api 1.7.1
#!/usr/bin/env…
wyx
- 3,334
- 6
- 24
- 44
0
votes
1 answer
Optionally use component functions added in VertexAI python SDK
I am using vertex ai's python SDK and it's built on top of Kubeflow pipelines. In it, you supposedly can do this:
train_op = (sklearn_classification_train(
train_data = data_op.outputs['train_out']
).
…
Optimus
- 1,354
- 1
- 21
- 40
0
votes
1 answer
issues when setting up kubeflow pipeline on minikube
I have a minikube running on macos. When trying to setup kubeflow pipeline I got the following output:
(base) ~/ml $ export PIPELINE_VERSION=1.7.0
(base) ~/ml $ kubectl apply -k…
Charles Ju
- 1,095
- 1
- 9
- 28
0
votes
1 answer
Kubeflow - Job Finished executing but is still running
My Kubeflow pipeline components/jobs continue to run indefinitely even though the main execution has finished. From these logs, might folks see why the job won't finish successfully?
It seems that there is a wait container that continues to run,…
ashemag
- 86
- 4