Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
0
votes
0 answers

How to get logs from runtime packages_to_install in python kfp.v2.dsl.component running on Vertex AI?

When running a kfp pipeline with custom components (python function wrappers) that use a base image and packages_to_install on top of that, the component may silently fail without any descriptive error. from kfp.v2.dsl import component @component( …
Proxitrone
  • 1
0
votes
0 answers

Delete attempt of Kubernetes resource reports not found, even though it can be listed with "kubectl get"

I am running Kubeflow pipeline on a single node Rancher K3S cluster. Katib is deployed to create training jobs (Kind: TFJob) along with experiments (a CRD). I can list the experiment resources with kubectl get experiments -n . However,…
Mark S
  • 1
0
votes
0 answers

How can I access a VM in a different subnet from a function-based Kubeflow component within Google Cloud?

I recently have been working with Kubeflow Pipelines on Google Cloud Platform. For my purposes the usage of function-based component specification is preferred. My goal is to access a Compute Engine VM that is outside of the subnet that the Notebook…
Just_Relax
  • 3
  • 3
0
votes
0 answers

Use existing resources for a new Kustomize installation? (kubeflow)

I am trying to install kubeflow pipelines (KFP) for kubeflow on AWS, as shown here. I am using an overlay for some simple labeling and other cosmetic changes. Installing KFP in the way shown in the documentation will also deploy instances of argo…
Xela
  • 71
  • 8
0
votes
0 answers

How to process SIGTERM in argo or kubeflow stage/node/component?

How to process SIGTERM in argo or kubeflow stage/node/component? It's possible to catch SIGTERM if your python script launched with PID 1. But in argo/kubeflow container PID 1 is occupied by 1 root 0:00 /var/run/argo/argoexec emissary -- bash…
ice1x
  • 21
  • 6
0
votes
2 answers

How to make a kubeflow pipeline step depend on multiple previous steps

I am running several kf steps in parallel. When they all complete AND if they have all succeeded I would like to trigger a last final step. With my current implementation the last step triggers if any of the previous ones succeeds which is not what…
G. Macia
  • 1,204
  • 3
  • 23
  • 38
0
votes
1 answer

Vertex AI Pipelines (Kubeflow) skip step with dependent outputs on later step

I’m trying to run a Vertex AI Pipelines job where I skip a certain pipeline step if the value of a certain pipeline parameter (in this case do_task1) is False. But because there is another step that runs unconditionally and expects the output of the…
VanDough
  • 21
  • 3
0
votes
0 answers

Kubeflow pipelines: Component stuck

I am trying to run probably the most basic kubeflow pipeline as described at this link: https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/ The Pipeline is just calling an "add" function 2 times inside a dsl pipelines.…
Himanshu Gautam
  • 359
  • 1
  • 4
  • 17
0
votes
1 answer

Failed to authorize with Kubeflow API resource references: PermissionDenied: User 'namespace'

I am trying to build a sample pipeline of adding two numbers. While running kfp.Client().create_run_from_pipeline_func(calc_pipeline, arguments=arguments) below error I a getting for faceless account. **ApiException: (403) Reason: Forbidden HTTP…
Vinay Kumar
  • 1
  • 1
0
votes
1 answer

how do we restore kubeflow from backups if the installation is destroyed or how we can back the kubflow as it was if the eks cluster is destroyed

How am i going to take backup for my kubeflow pipeline and restore it if the installing is failed or the eks cluster is destroyed. i have some finding to get the image of the vanila i am using for database and find out how to take backup and restore…
sabbir ahmed
  • 19
  • 2
0
votes
1 answer

Kuveflow volumens lost data

Hello I am trying to share file between steps, and In order to do this I have the following code: VOLUME_NAME_PATH = 'pictures' VOLUME_PATH = f'/{VOLUME_NAME_PATH}' V1_VOLUME = k8s_client.V1Volume(name=VOLUME_NAME_PATH) V1_VOLUME_MOUNT =…
Tlaloc-ES
  • 4,825
  • 7
  • 38
  • 84
0
votes
2 answers

How can I use a lot pictures in a kubeflow pipeline?

Hello I am training a yolo in a kubeflow pipeline, in order to this, I have a set of pictures more than 1GB. Currently, I download all images from minio to the container with a script and after that I train the model. I am not sure if is there any…
Tlaloc-ES
  • 4,825
  • 7
  • 38
  • 84
0
votes
1 answer

Pod "no2-pipeline-x5kpd-2954674781" is invalid: spec.volumes[3].name: Duplicate value: "no2-pvc"

Hi I am trying to run a Kubeflow pipeline. Two steps will run in parallel and dump data to two different folders of PVC, then the third component will collect data from those to folders and merge them together and dump the merged data to another PVC…
Md Hafizur Rahman
  • 21
  • 4
0
votes
2 answers

Kubeflow Pipeline Training Component Failing | Unknown return type:

I am running an ML pipeline and the training component/step (see code below) continues to fail with the following error: "RuntimeError: Unknown return type: . Must be one of str, int, float, a subclass of Artifact, or a…
RE Wolfe
  • 29
  • 3
0
votes
1 answer

Pods not running with an error message of IMAGEPULLBACKOFF

I initialized kubeflow pods using the following command. juju deploy kubeflow The following two pods didn't run and gave an error message stating IMAGEPULLBACKOFF. kfp-viz, kfp-profile-controller Yaml code for kfp-viz Name: …
Soumil Maitra
  • 11
  • 3
1 2 3
…
18 19