Stack Overflow
Stack Overflow

Questions tagged [kubeflow]

Kubeflow Training Operator provides Kubernetes custom resources that makes it easy to run distributed or non-distributed TensorFlow/PyTorch/Apache MXNet/XGBoost/MPI jobs on Kubernetes.

GitHub: https://github.com/kubeflow/training-operator

433 questions
5
votes
3 answers

What is the difference between GCP Kubeflow and GCP cloud composer?

I am learning GCP, and came across Kuberflow and Google Cloud Composer. From what I have understood, it seems that both are used to orchestrate workflows, empowering the user to schedule and monitor pipelines in the GCP. The only difference that I…
Nizam
  • 340
  • 1
  • 6
  • 11
5
votes
0 answers

How to fix PipelineParam from discarding all information except for name in Kubeflow Pipeline

I'm trying to write an application using Kubeflow Pipelines. I'm running into trouble when passing in parameters to the pipeline (the main python function decorated with @kfp.dsl.pipeline). The parameters should be automatically converted into a…
Nicholas Schenone
  • 51
  • 3
4
votes
0 answers

download from URL Kubeflow pipeline

I have been trying to create a kubeflow pipeline in my local machine and created a simple one component pipeline to download data from a given url. import kfp import kfp.components as comp downloader_op =…
user395882
  • 665
  • 2
  • 7
  • 16
4
votes
1 answer

Kubeflow - how to use if else statement in pipeline?

I need to make a pipeline to create a dataset for training a model. I am using two data sources. I would like to be able to decide whether to use one source, the other, or both. To do this, I have created two pipeline parameters, for…
nietoperz21
  • 303
  • 3
  • 12
4
votes
1 answer

How to skip already-run steps in kubeflow pipeline?

I'm building an ML pipeline in Kubeflow and I have a question. Is there anything out of the box that allows me to configure my pipeline, such that a step is not rerun if its output exists? I've thought of ways to do this manually (either checking…
Elisabeth Guegan
  • 109
  • 1
  • 5
3
votes
0 answers

Access information from a previous Kubeflow component

I have a ModelBatchPredictOp component in my pipeline. This component generates 3 artifacts: batchpredictionjob, big_query_table, and gcs_output_directory. The pipeline is running fine. What I need is a way to access the tableId property of artifact…
Openworld
  • 31
  • 3
3
votes
0 answers

KUBEFLOW: no such file or directory Error

I am creating a very simple kubeflow pipeline which has the following steps: Data Loading Data Preprocessing Training Evaluation The first 3 steps runs successfully but as soon as I try to run the evaluator the step fails with the error mentioned…
R.Ahuja
  • 51
  • 4
3
votes
0 answers

Kubeflow installation on existing EKS cluster with cert-manager already installed

I have an EKS cluster (1.18 version)in the pre-production environment. Now I would like to use this cluster to install Kubeflow (1.4 version). Unfortunately, when I try to install it with the kfctl apply -V -f kfctl_aws.yaml I get this…
UNix3
  • 31
  • 2
3
votes
1 answer

How can I programmatically create a Kubeflow recurring run from a pipeline function?

I am trying to create a recurring kubeflow pipeline run as follows: from kfp import compiler compiler.Compiler().compile( pipeline_func=my_pipeline, package_path='pipelines/my_pipeline.tgz') from kfp.v2.google.client import…
jtlz2
  • 7,700
  • 9
  • 64
  • 114
3
votes
1 answer

Upgrade /bin/bash on MacOS to v5+

I am trying to install Anthos Service Mesh (ASM) for a Kubeflow installation and need /bin/bash to be v5+. MacOS comes with Bash v3.2.57 which doesn't work. Simply installing Bash v5+ in "/usr/local/bin" doesn't work either as several shell scripts…
UlrikP
  • 412
  • 3
  • 8
3
votes
2 answers

Kubeflow pipeline fail to create container

I'm running Kubeflow in a local machine that I deployed with multipass using these steps but when I tried running my pipeline, it got stuck with the message ContainerCreating. When I ran kubectl describe pod train-pipeline-msmwc-1648946763 -n…
João Areias
  • 1,192
  • 11
  • 41
3
votes
1 answer

Get experiment name from within a Kubeflow pipeline run

I just started working with Kubeflow and I ran into a problem. I need my pipeline to be able to automatically get the name of the experiment it belongs to. I tried to use the kfp package but it seems to me that there is no way to get the experiment…
user2846
  • 133
  • 2
3
votes
2 answers

Connecting to AI Platform Pipelines using the Kubeflow Pipelines SDK with Python and PyCharm

Using windows 10 and Python 3.9 with PyCharm IDE I am trying to list the pipelines using the following code: import kfp client = kfp.Client(host='.......bb92ea05-dot-us-central2.pipelines.googleusercontent.com') client.list_pipelines() I know…
Franco
  • 441
  • 3
  • 18
3
votes
1 answer

Allocate resources for Kubeflow pipeline using pipeline params

I would like to be able to create a Kubeflow pipeline that allows users to set the allocated resources for a run. The end result would be something like this: Example of Kubeflow "Create Run" UI with ability to set resource allocation. Definition of…
Carter
  • 33
  • 3
3
votes
1 answer

Can I mount volume to Katib Experiment?

I am using the .yaml file below to create Katib Experiment in Kubeflow. However, I am getting Failed to reconcile: cannot restore struct from: string errors. Have any solutions for this? Most of the Katib Experiment example codes doesn't have a…
Piljae Chae
  • 987
  • 10
  • 23
1
2
3
…
28 29