Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
2
votes
1 answer

Read the output of a kubeflow pipeline programatically

I run pipelines on Kubeflow with a python command like: client.create_run_from_pipeline_func(pipeline_function, arguments=params_dict[name], run_name=name) It creates a job on Kubeflow pipelines and I would like to be able to access the information…
Robin Nicole
  • 646
  • 4
  • 17
2
votes
0 answers

Vertex AI Pipeline is not using the GPU

I am building a customized pipeline with the following step: trainer_task = (trainer(download_task.output).set_cpu_request("16").set_memory_request("60G"). add_node_selector_constraint('cloud.google.com/gke-accelerator',…
spalacio
  • 21
  • 1
2
votes
1 answer

Is it possible to output a list of artifacts of the same type using kubeflow pipelines?

I'm trying to output multiple confusion matrices from a kubeflow pipeline component, like in this example that only outputs one: def eval_model( test_set: Input[Dataset], xgb_model: Input[Model], metrics: Output[ClassificationMetrics], …
VanDough
  • 21
  • 3
2
votes
0 answers

How to use Kubeflow volume mount with outputPath parameter?

I am building a Kubeflow pipeline that has 2 components. Component 1 preprocesses some data and component 2 performs model training on that data. I understand I need to save the data at some outputPath parameter generated by Kubeflow. This works. I…
Zach
  • 113
  • 1
  • 9
2
votes
0 answers

Kubeflow: Notebook server stuck on loading

Whenever I try to create a Kubeflow notebook server to build a pipeline from a jupyter notebook, it keeps loading forever without displaying any error. I'm currently using a Kubeflow dashboard that's already up and running on a server, so I didn't…
camelia
  • 41
  • 4
2
votes
1 answer

AWS Managed Airflow vs. AWS Lambda + Step Functions vs. Kubeflow on AWS EKS

This is going to be a fairly general question. I have a pipeline that I would like to execute in real time. The pipeline can have sudden and unpredictable load changes, so scalability (both up and down) are important. The pipeline stages can be…
bumpbump
  • 542
  • 4
  • 17
2
votes
3 answers

Kubeflow - error in create_run_from_pipeline_func

I'm new to Kubeflow and k8s. I have setup a single node k8s cluster and installed Kubeflow on this. I'm now trying the 'conditional pipeline' simple example from "Kubeflow for Machine Learning" book but I am getting "cannot post…
soumeng78
  • 600
  • 7
  • 12
2
votes
2 answers

What is the best option for build kubeflow components?

I am read about Kubeflow, and for create components there are two ways. Container-Based Function-Based But there isn't an explication about why I should to use one or another, for example for load a Container-based, I need to generate a docker…
Tlaloc-ES
  • 4,825
  • 7
  • 38
  • 84
2
votes
0 answers

Is it possible to use artifacts as source for visualisations in Kubeflow pipelines

I'm experimenting with Kubeflow on minikube and I try to use the visualizations feature of the Kubeflow pipeline UI. The documentation states that you should generate a mlpipeline-ui-metadata.json file and add it to the ContainerOp outputs. This…
alberthier
  • 643
  • 1
  • 7
  • 9
2
votes
1 answer

Instantiate and Shutdown Kubeflow pods

I'm learning about Kubernetes and Kubeflow, and there's something that I want to do that I'm not finding any clear answer on the internet on if it's possible or the route I should take. When training my machine learning model, I want to use a large…
João Areias
  • 1,192
  • 11
  • 41
2
votes
1 answer

Deploying Jupyter notebooks as a component in AI Platform pipelines

I have a Jupyter Notebook containing the model creation, model deployment on the AI Platform, and creating versions. I am able to get the predictions for my model. Now I am trying to build a CI/CD pipeline automating the entire process. Is there a…
Hemanth Ponnada
  • 71
  • 1
  • 5
2
votes
2 answers

Compilation of Elyra-Pipelines to Tekton based Kubeflow fails

I've installed a kubernetes cluster running kubeflow pipelines based on tekton on top of KIND using the following instructions Now I'm getting the following error message from the Elyra pipelines editor. Running against an argo based kfp cluster…
Romeo Kienzler
  • 3,373
  • 3
  • 36
  • 58
2
votes
1 answer

JupyterLab/Elyra: pipeline run on Kubeflow Pipelines fails with "No host specified" in local deployment

I have Kubeflow Pipelines running in my local environment, along with JupyterLab and the Elyra extensions. I've created a notebook pipeline and configured the runtime configuration as follows, setting api_endpoint to http://localhost:31380/pipeline…
ptitzler
  • 923
  • 4
  • 8
2
votes
2 answers

ParallelFor in Kubeflow Pipelines

I'd like to use a custom list to run parallel Ops in a Kubeflow Pipeline, and I want to use the value of the element of the list into the definition of the Op. I'm trying something like this: my_list = ['foo', 'bar'] with dsl.ParallelFor(my_list) as…
Matteo Felici
  • 1,037
  • 10
  • 19
2
votes
1 answer

How do I implement the Kubeflow "Run Paramters" with the TFX SDK specialized for GCP?

I am currently using Kubeflow as my orchestrator. The orchestrator is actually an instance of an AI platform pipeline hosted on GCP. How do I create run-time parameters using the Tensorflow Extended SDK? I suspect that this is the class that I…
coderhk
  • 286
  • 2
  • 14
1 2 3
…
18 19