Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
0
votes
0 answers

Kubeflow pipeline Exit Handler with multiple components/steps

I'm trying to configure an Exit Handler in Kubeflow Pipelines to use two different components that perform two distinct actions at the end of the pipeline run. I've tried to do it in so many different ways, mostly using trial and error, as I've not…
Albert
  • 1
0
votes
1 answer

Configure subnetwork for vertex ai pipeline component

I have a vertex ai pipeline component that needs to connect to a database. This database exists in a VPC network. Currently my component is failing because it is not able to connect to the database, but I believe I can get it to work if I can…
DarioB
  • 1,349
  • 2
  • 21
  • 44
0
votes
1 answer

Migrate kubeflow docker image components to VertexAI pipeline

I am trying to migrate a custom component created in kubeflow to VertexAI. In Kubeflow I used to create components as docker container images and then load them into my pipeline as follows: def my_custom_component_op(gcs_dataset_path: str,…
DarioB
  • 1,349
  • 2
  • 21
  • 44
0
votes
0 answers

Vertex AI tutorial error ; AiPlatformException: code=RESOURCE_EXHAUSTED

I have a trouble about sample code in https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_train_upload_deploy.ipynb. I tried notebook for vertex AI pipeline however…
ArataTakahashi
  • 1
0
votes
0 answers

Match individual records during Batch predictions with VertexAI pipeline

I have a custom model in Vertex AI and a table storing the features for the model along with the record_id. I am building pipeline component for the batch prediction and facing a critical issue. When I submit the batch_prediction, I should exclude…
CHOCOLEO
  • 363
  • 2
  • 9
0
votes
0 answers

Kubeflow Parallel For how to report progress

In Kubeflow, how can I log the progress of the for loop to see if it is progressing? Moreover, in the graph view, the graph renderer is having a hard time to draw 1000s of tasks. Can it be grouped?
Hanan Shteingart
  • 8,480
  • 10
  • 53
  • 66
0
votes
2 answers

Kubeflow parallel for loop is ignoring `parallelism` parameter

There is not good documentation for dsl.ParallelFor but I assume the parallelism param means - how many pods can be opened in parallel. However, when I look at the DAG visualization, it seems like it opens all the for loop list of tasks, resulting…
Hanan Shteingart
  • 8,480
  • 10
  • 53
  • 66
0
votes
1 answer

Inconsistent behaviour of Kubeflow ParallelFor in Vertex AI

I have a Vertex A pipeline that I use for example: run_1_output = run_1().set_memory_limit("1G").set_cpu_limit("1") with kfp.dsl.ParallelFor(run_1_output.output) as key: run_1a_output = run_1a(key=key).set_memory_limit("1G").set_cpu_limit("1") …
Martin Becuwe
  • 117
  • 2
  • 9
0
votes
1 answer

Basic question on downloading data for a Kubeflow pipeline

I'm a newbie on Kubeflow, just started exploring. I've setup a microk8s cluster and charmed kubeflow. I have executed a few examples trying to understand the different components. Now I'm trying to setup a pipeline from scratch for a classification…
Govi
  • 1
0
votes
0 answers

KFP define how to manipulate the PipelineArgs during pipeline build

Use case: Use Airflow operators to launch DataflowTemplate from KFP pipelines but the pipeline should be parameterised. Issue: I cant find a way in which I can pass the KFP pipeline args to the component function as extra args. i.e., import os from…
Tameem
  • 408
  • 7
  • 19
0
votes
0 answers

Passing model uri to seldon core deployment in kubeflow (locally)

I am trying to pass model uri which is stored in kubeflow component as an output path. To a seldon core deployment That's the code which train's model and outputh's file path as a joblib file. According to seldon core documentation to pass a model…
minio999
  • 3
  • 2
0
votes
0 answers

How to return a boolean parameter in a Kubeflow component along with a dataset

Im developing a set of kubeflow lightweight components. I know that, if all I want to do is return a bool I can do it this way: def mockup_data_drifting_comp(output_component_file): @component( packages_to_install=........., …
Sergiodiaz53
  • 1,268
  • 2
  • 14
  • 23
0
votes
0 answers

Vertex AI (Kubeflow) pipeline doesn't pick-up metrics from file /mlpipeline-metrics.json

I am testing the metrics display in Vertex (Kubeflow UI for a pipeline running on GCP. I use the reusable component method with component specified in YAML, not function based components. The component is containerized and pushed to artifact…
crbl
  • 379
  • 2
  • 13
0
votes
0 answers

How to run Kubeflow Pipelines via HTTP API?

We have a hosted instance of Kubeflow that I am trying to access programmatically. Although Kubeflow v1 has Swagger docs for an HTTP API, v2 does not. There's almost no mention of authentication, let alone actual descriptions or examples of how to…
Everett
  • 8,746
  • 5
  • 35
  • 49
0
votes
1 answer

passing data between Kubeflow pipelines containers

So I am wondering how can I pass data frame then model between Kubeflow containers which I deployed locally using k8s. Right now I am using v2 sdk and func_to_container_op decorator but I am also interested in how to do it using docker files and…
minio999
  • 3
  • 2
1 2 3
…
18 19