Questions related to the kubeflow pipelines python SDK
The kubeflow pipelines python SDK kfp is used to programmatically interact with kubeflow pipelines and the underlying kubernetes cluster.
Questions tagged [kfp]
34 questions
1
vote
2 answers
How to use ExitHandler with Kubeflow Pipelines SDK v2
I'm trying to move all my Kubeflow Pipelines from using the previous SDK v1 (kfp), to the newer Pipelines SDK v2 (kfp.v2). I'm using version 1.8.12.This refactoring have proved successful for almost all code, except for the ExitHandler, which still…
Marcus
- 943
- 5
- 21
1
vote
1 answer
Add decorator to component decorator in KFP v2 in Vertex AI
Usually, KFP v2 supports adding a component decorator like this:
@component
def test():
print("hello world")
I would like to add an additional decorator to add new functionality like this:
@component
@added_functionality
def test():
…
Optimus
- 1,354
- 1
- 21
- 40
1
vote
2 answers
VertexAI Pipeline: How to use an output from a custom kfp component as input for google_cloud_pipeline_components?
I'm trying to write the Python code for a pipeline in VertexAI using kfp components. I have a step where i create a system.Dataset object that is the following:
@component(base_image="python:3.9",…
Marco Abbatangelo
- 77
- 1
- 11
1
vote
1 answer
OSError: [WinError 123] when I create kfp component
I'm trying to create a pipeline in Vertex AI with kfp using my own components from local notebook in Spyder.
When I run the following piece of code:
@component(base_image="python:3.9", packages_to_install=["pandas"])
def create_dataset(
…
Marco Abbatangelo
- 77
- 1
- 11
1
vote
2 answers
google's notebook on vertex ai throwing following error: type name google.VertexModel is different from expected: Model
I got this error, when compiling my pipeline:
type name google.VertexModel is different from expected: Model
when running the following notebook by google: automl_tabular_classification_beans
I suppose that kubeflow v2 is not able to handle (yet)…
EiJay
- 43
- 5
0
votes
0 answers
Always 404 on kfp 2.0.1
I was using kfp==2.0.0b12 and everything was working great. But as soon as i got into 2.0.1 I am always getting 404 when trying to use methods from client, like this:
kfp_server_api.exceptions.ApiException: (404)
Reason: Not Found
HTTP response…
d0m3n1
- 1
- 1
0
votes
1 answer
How do I get Vertex AI pipeline output?
I define a Vertex AI pipeline thus:
@dsl.pipeline
def pipe_pipeline(arg1: str, arg2: str) -> str:
...
compile it:
compiler.Compiler().compile(pipe_pipeline, 'pipeline.yaml')
and submit it:
ml_pipeline_job = aiplatform.PipelineJob(
…
schoon
- 2,858
- 3
- 46
- 78
0
votes
1 answer
Enable web access or interactive shell for PipelineJob tasks for Vertex AI
I am trying to debug a PipelineJob that I launch on Vertex AI. Is there a way to enable web access on the components like you can when you launch Custom Jobs? This way, I could ssh into the running task and do a bunch of debugging.
Here is a…
Optimus
- 1,354
- 1
- 21
- 40
0
votes
1 answer
Get the image name inside KFP component definition
I have a python function that is converted to KFP component using @component decorator, like this:
@component(
base_image="image_name:image_tag",
)
def func(arg1):
# I want to get the image used here, e.g. I want to get image =…
eng2019
- 953
- 10
- 26
0
votes
0 answers
KFP define how to manipulate the PipelineArgs during pipeline build
Use case:
Use Airflow operators to launch DataflowTemplate from KFP pipelines but the pipeline should be parameterised.
Issue:
I cant find a way in which I can pass the KFP pipeline args to the component function as extra args. i.e.,
import os
from…
Tameem
- 408
- 7
- 19
0
votes
1 answer
passing data between Kubeflow pipelines containers
So I am wondering how can I pass data frame then model between Kubeflow containers which I deployed locally using k8s. Right now I am using v2 sdk and func_to_container_op decorator but I am also interested in how to do it using docker files and…
minio999
- 3
- 2
0
votes
0 answers
How to get logs from runtime packages_to_install in python kfp.v2.dsl.component running on Vertex AI?
When running a kfp pipeline with custom components (python function wrappers) that use a base image and packages_to_install on top of that, the component may silently fail without any descriptive error.
from kfp.v2.dsl import component
@component(
…
0
votes
1 answer
How do I undeploy a model from an endpoint without knowing its id in Vertex AI?
I have managed to undeploy a model from an endpoint using UndeployModelRequest:
model_name = f'projects/{project}/locations/{location}/models/{model_id}'
model_request = aiplatform_v1.types.GetModelRequest(name=model_name)
model_info =…
schoon
- 2,858
- 3
- 46
- 78
0
votes
0 answers
Why can't I access Output from Vertex pipeline kfp component?
In a Vertex AI pipeline (google_cloud_pipeline_components version: 1.0.19
kfp version: 1.8.13), I try:
does_endpoint_exist_op = does_endpoint_exist(project=project,
location=location, endpoint_name_in=endpoint_name)…
schoon
- 2,858
- 3
- 46
- 78
0
votes
1 answer
Why does my dbt container hang in Vertex AI?
I am trying to follow this tutorial to run a dbt docker image as a Vertex AI component. When the pipeline runs the component just seems to sit there for ever. Is there any way of debugging the component?
schoon
- 2,858
- 3
- 46
- 78