Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
0
votes
2 answers

Equivalent of TFX Standard Components in KubeFlow

I have an existing TFX pipeline here that I want to rewrite using the KubeFlow Pipelines SDK. The existing pipeline is using many TFX Standard Components such as ExampleValidator. When checking the KubeFlow SDK, I see a kfp.components.package but no…
pirateofebay
  • 930
  • 1
  • 10
  • 25
0
votes
2 answers

Serving custom model from Kubeflow pipeline

I have a kubeflow pipeline which trains custom (i.e. not based on sklearn / tensorflow etc. classes) ml model. Now I would like to add serving at the end of the pipeline. I.e. I want to have a service in my Kubernetes cluster which uses the model to…
Tomasz Cakala
  • 1
  • 1
0
votes
1 answer

How to change parameters for a Kubeflow Pipeline Recurring Run?

I would like to run my Kubeflow pipeline every 30 minutes using a "Recurring Run". For each run I would like to tweak my parameters. For example: First run : START_MONTH=2019-01,END_MONTH=2010-02 Second run :…
sanj2sanj
  • 71
  • 6
0
votes
1 answer

How to use pipeline for Apache Spark jobs?

I’m learning how to use kubeflow pipeline for Apache Spark jobs and have a question. I’d appreciate if you could share your thoughts! It is my understanding that data cannot be shared between SparkSessions, and that in each pipeline step/component…
lollicoco
  • 13
  • 2
0
votes
1 answer

Using kfp.dls.containerOp() to run multiple scripts on Kubeflow Pipelines

I have been using the Kubeflow dsl container op command to run a python script on a custom for my Kubeflow pipeline. My configuration looks something like this : def test_container_op(): input_path = '/home/jovyan/' return dsl.ContainerOp( …
red_devil
  • 1,009
  • 2
  • 13
  • 23
0
votes
1 answer

How to log metrics from component using YAML specification?

We are using kpf v2 in Google Clouds Vertex AI and want to log metrics from a component built by a yaml specification. We can only make this work from a Python function based component were we can call the mlpipeline-metrics (of type…
sniggatooth
  • 131
  • 3
0
votes
2 answers

How to track parameter and metrics from Vertex AI pipelines

We are using both Vertex AI training jobs and Kubeflow pipelines in Google Clouds Vertex AI. In training jobs we log parameters and metrics to Vertex AI Experiments through the python sdk. Can Vertex AI Pipelines track metrics from the Kubeflow…
sniggatooth
  • 131
  • 3
0
votes
1 answer

How can I run a pipeline when a file is uploaded to a GCP bucket (Kubeflow 1.3 multi-user)?

I get the following error: "Failed to authorize with API resource references: Failed to authorize with API resource references: PermissionDenied: User 'myServiceAccount@appspot.gserviceaccount.com' is not authorized with reason: (request:…
pablofiumara
  • 1,747
  • 1
  • 16
  • 24
0
votes
1 answer

Why pod on GKE cluster is OOMkilled when trying to run a very simple Kubeflow pipeline using TFX?

I'm following the TFX on Cloud AI Platform Pipelines tutorial to implement a Kubeflow orchestrated pipeline on Google Cloud. The main difference is that I'm trying to implement an Object Detection solution instead of the Taxi application proposed by…
A. Longo
  • 1
  • 4
0
votes
1 answer

ipython terminates with 'Syntax error: end of file unexpected' inside Kubeflow Pod, not happening in Docker container

I'm trying to create a Kubeflow Pipeline component. My command works fine in a local Docker container, but within a Pod in a Kubeflow Pipeline it terminates with: ipython /tmp/input-postgresql.ipynb: 1: ipython /tmp/input-postgresql.ipynb: Syntax…
Romeo Kienzler
  • 3,373
  • 3
  • 36
  • 58
0
votes
1 answer

Example notebook, sample for google cloud pipeline component

I am looking for a sample or a tutorial notebook with specifically the "CustomPythonPackageTrainingJobRunOp" google cloud pipeline component. I have been trying to use this but keep getting into errors. PS: I have already posted a question about the…
racerX
  • 930
  • 9
  • 25
0
votes
1 answer

Working link for Google Cloud pipeline components docs?

Does anyone have a working link for the docs for Google Cloud pipeline components. The link in the github page under "ReadTheDocs page" is broken. Tried some other tutorial notebooks, such as this one, the link under "The components are documented…
racerX
  • 930
  • 9
  • 25
0
votes
2 answers

Kubeflow sdk - error in client.list_experiments()

I'm following the example in docs (https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/) but am getting an error when trying to access the experiments or runs (when trying to list and/or create them) I've port-forwarded…
Tanja du Plessis
  • 21
  • 5
0
votes
3 answers

how to achieve multi-user isolation for kubeflow pipelines

I recently updated kubeflow version to 1.3 but this does not seem to work for API authentication. When I call kubeflow endpoint from my springboot service, it gives below error nested exception is…
esha ingle
  • 163
  • 1
  • 3
  • 18
0
votes
1 answer

How to parameterize Kubeflow Pipelines environment variables?

I am exploring Vertex AI Pipelines for running machine learning training jobs. The kubeflow pipeline docs are clear about how to parameterize the commands/arguments of a container. Is it also possible to pass a input to an environmental variable or…
nbren12
  • 634
  • 1
  • 7
  • 11
1 2 3
…
18 19