How to persist variables in google cloudbuild steps?

Question

I have a cloudbuild.json which is used to upload a pipeline to gcp kubeflow. now i want to add another step in which i want to fetch the latest pipeline id and then run the pipeline as an experiment. so my main issue is how should i get the pipeline id in the subsequent steps. i have written a small script to fetch the latest pipeline id and also add it as a step running from docker but now i am not sure how would i fetch this pipeline id.

here is my coudbuild.json

{
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "trainer_image",
                "."
            ],
            "dir": "./trainer_image/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "base_image",
                "."
            ],
            "dir": "./base_image/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
            ],
            "env": [
                "BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test",
                "TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
                "RUNTIME_VERSION=1.15",
                "PYTHON_VERSION=3.7",
                "COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/",
                "USE_KFP_SA=False"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "id_image",
                "."
            ],
            "dir": "./id_image/"
        }
    ],
    "images": [
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
    ]
}

here is my python script to fetch latest pipeline id

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
    file.write(latest_pipeline_id)

score 2 · Answer 1 · answered Sep 11 '20 at 14:01

You can't persist a variable between step. You can only persist files. However inside a step, you can reuse local variable.

- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable

How to persist variables in google cloudbuild steps?

1 Answers1