0

We have some services that can be installed in multiple locations with differing configurations. We've been asked to support multi-level configuration options using environment variables set with defaults, configmaps, secrets, and command-line options passed in via helm install --set. The following works, but is very cumbersome as the number of parameters for some of the services are numerous and the Values dot-notation goes a few levels deeper.

env:
  # Set default values
  - name: MY_VAR
    value: default-value
  - name: OTHER_VAR
    value: default-other-value

  # Allow configmap to override
  - name: MY_VAR
    valueFrom:
      configMapKeyRef:
        name: env-configmap
        key: MY_VAR
        optional: true
  - name: OTHER_VAR
    valueFrom:
      configMapKeyRef:
        name: env-configmap
        key: OTHER_VAR
        optional: true

  # Allow secrets to override
  - name: MY_VAR
    valueFrom:
      secretsKeyRef:
        name: env-secrets
        key: MY_VAR
        optional: true
  - name: OTHER_VAR
    valueFrom:
      secretsKeyRef:
        name: env-secrets
        key: OTHER_VAR
        optional: true

  # Allow 'helm install --set' to override
  {{- if .Values.env }}
  {{- if .Values.env.my }}
  {{- if .Values.env.my.var }}
  - name: MY_VAR
    value: {{ .Values.env.my.var }}
  {{- end }}
  {{- end }}
  {{- if .Values.env.other }}
  {{- if .Values.env.other.var }}
  - name: OTHER_VAR
    value: {{ .Values.env.other.var }}
  {{- end }}
  {{- end }}
  {{- end }}

Using envFrom for the ConfigMap and Secrets would be nice, but tests and docs show this would not allow the command-line override, since env: and envFrom: doesn't mix in the way that's needed. As the v1.9 and v2.1 Kubernetes API states:

envFrom: List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated.

Is there a better way to provide this default->configmap->secrets->cmd-line override precedence?

Tog
  • 11
  • 3

1 Answers1

1

I found a solution that I mostly like. My issue was caused by giving too much weight to the "Values defined by an Env with a duplicate key will take precedence" comment in the docs, and thinking I needed to exclusively use Env. The defined precedence is exactly what I needed.

Here's the helm chart files for my current solution.

configmap/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  {{- if .Values.env }}
    {{- toYaml $.Values.env | nindent 2 }}
  {{- end }}
  {{- if .Values.applicationYaml }}
  application.yml: |
    {{- toYaml $.Values.applicationYaml | nindent 4 }}
  {{- end }}

secrets/templates/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: my-secrets
type: Opaque
data:
  {{- range $key, $val := .Values.env }}
  {{ $key }}: {{ $val | b64enc }}
  {{- end }}
stringData:
  {{- if .Values.applicationYaml }}
  application.yml: |
    {{- toYaml $.Values.applicationYaml | nindent 4 }}
  {{- end }}

deployment.yaml

apiVersion: apps/v1
kind: Deployment
  ...
spec:
  ...
      containers:                                                                                     
      - name: my-deployment
        {{- if .Values.env }}
        env:
        {{- range $key, $val := .Values.env }}
        - name: {{ $key }}
          value: {{ $val }}
        {{- end }}
        {{- end }}
        envFrom:
        - configMapRef:
           name: my-configmap
        - secretRef:
           name: my-secrets
        volumeMounts:
        - name: configmap-application-config
          mountPath: /application/config/configmap/
        - name: secrets-application-config
          mountPath: /application/config/secrets/
      volumes:
        - name: configmap-application-config
          configMap:
             name: my-configmap
             optional: true
        - name: secrets-application-config
          secret:
             secretName: my-secrets
             optional: true

Since this is a Spring Boot app, I used volumeMounts to allow the application.yml default values to be overridden in the ConfigMap and Secrets. The order of precedence from lowest to highest is:

  • the application's application.yml (v1 in following examples)
  • the configmap's applicationYaml (v2)
  • the secret's applicationYaml (v3)
  • the configmap env (v4)
  • the secret env (v5)
  • the helm install/uninstall --set (v6)

To complete the example, here's test values yaml files and the command-line.

app/src/main/resources/application.yml

applicationYaml:
  test:
    v1: set-from-this-value
    v2: overridden
    v3: overridden
    v4: overridden
    v5: overridden
    v6: overridden

configmap/values.yaml

applicationYaml:
  test:
    v2: set-from-this-value
    v3: overridden
    v4: overridden
    v5: overridden
    v6: overridden

env:
   TEST_V4: set-from-this-value
   TEST_V5: overridden
   TEST_V6: overridden

secrets/values.yaml

applicationYaml:
  test:
    v3: set-from-this-value
    v4: overridden
    v5: overridden
    v6: overridden

env:
   TEST_V5: set-from-this-value
   TEST_V6: overridden

command-line

helm install --set env.TEST_V6=set-from-this-value ...

Ideally, I'd like to be able to use dot-notation instead of TEST_V6 in the env and --set fields, but I'm not finding a way in helm to operate only on the leaves of yaml. In other words, I'd like something like range $key, $val, but where the key is equal to "test.v6". If that was possible, the key could be internally converted to an environment variable name with {{ $key | upper | replace "-" "_" | replace "." "_" }}.

Tog
  • 11
  • 3