How to pass an environmental variable in kubeflow pipeline?

Question

I want the variable to be accessed by gcr.io/******/serve_model:lat5 Image which is an argument of gcr.io/******/deployservice:lat2

Initially I have tried passing the variable as argument but it didn't work, so I am trying to pass it as an environmental variable.
My environmental variable will be an url of GCP storage bucket from where my serve_model will access the .sav model file.

        name='web-ui',
        image='gcr.io/******/deployservice:lat2',
        arguments=[
        '--image', 'gcr.io/******/serve_model:lat5',
        '--name', 'web-ui',
        '--container-port', '8080',
        '--service-port', '80',
        '--service-type', "LoadBalancer"
        ]
        ).add_env_variable(V1EnvVar(name='modelurl', value=Model_Path))

Do you want to pass only one value? Did you considered `ConfigMap` to do it? https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ — PjoterS, Dec 03 '19 at 15:55
@PjoterS I pass my environmental variable in my pipeline python file as : ```web_ui.container.add_env_variable(V1EnvVar(name='modelurl', value=Model_Path))``` And have put the below in my .sh file to run in kubeflow(kubernetes): ```kubectl create configmap modelurl --from-literal=modelurl=Model_Path``` Is it a right way ? — harish kumaar, Dec 05 '19 at 05:01

santiago92 · Answer 1 · 2019-12-06T09:25:02.210

add_env_variable() is a function of a Container object that's exposed as a property of a ContainerOp.

So something like below would work. Refer the kfp dsl code here

model_path = 'gcp://dummy-url'
container_op = ContainerOp(name='web-ui',
                               image='gcr.io/******/deployservice:lat2',
                               arguments=[
                                   '--image', 'gcr.io/******/serve_model:lat5',
                                   '--name', 'web-ui',
                                   '--container-port', '8080',
                                   '--service-port', '80',
                                   '--service-type', "LoadBalancer"]
                               )
container_op.container.add_env_variable(V1EnvVar(name='model_url', value=model_path))

You can verify this by checking the YAML in the zip for the env section under -container

  - container:
      args:
      - --image
      - gcr.io/******/serve_model:lat5
      - --name
      - web-ui
      - --container-port
      - '8080'
      - --service-port
      - '80'
      - --service-type
      - LoadBalancer
      env:
      - name: modelurl
        value: gcp://dummy-url <--the static env value
      image: gcr.io/******/deployservice:lat2

It seems that already used this but didn't paste whole code in question. — PjoterS, Dec 05 '19 at 15:03
The code that's posted in the question is running `add_env_variable()` right after the `)` which makes it seem like you're running it on the `ContainerOp` object, and *not* the `Container` object nested inside. Running the above code should get you the URL in the env (check the yaml snippet above). If your yaml shows like that, the env var should be accessible in the pipeline. My point being, if the URL path is a static string, you do not need a ConfigMap for it. — santiago92, Dec 06 '19 at 09:28

score 0 · Accepted Answer · answered Dec 05 '19 at 14:53

Posting this as Community Wiki for better visibility as Original Poster was able to pass this variable.

It's the best Kubernetes way to pass value.

ConfigMap is a dictionary of configuration settings. This dictionary consists of key-value pairs of strings. Kubernetes provides these values to your containers. ConfigMap stores configuration settings for your code. Store connection strings, public credentials, hostnames, and URLs in your ConfigMap.

You can create ConfigMap in many ways (from file, manually, etc). More information can be found here.

Solution

According to Original Poster comment:

1. Pass environmental variable using pipeline python file and container function add_env_variable:

web_ui.container.add_env_variable(V1EnvVar(name='modelurl', value=Model_Path))

2. Prepare command which will create config map with proper value:

kubectl create configmap modelurl --from-literal=modelurl=Model_Path

3. Put previous command to script which will be used in Kubeflow.

How to pass an environmental variable in kubeflow pipeline?

2 Answers2