generate dynamic runtime YAML for container with k8s secrets

Question

I'm deploying a service which rigidly takes a YAML config that looks something like this:

# /config/srv.yaml

#
# ...
#
tenantid: '...'
clientid: '...'
certfile: '/config/client/client.crt'
keyfile:  '/config/client/client.key'
#
# ...
#

TL;DR; I want to only store the tenantid, clientid & cert files in k8s secrets.

My current solution, stores the entire YAML file in secrets, which seems wasteful and also cumbersome to manage.

Essentially I have this - which works well for the 2 cert files (/config/client/client.crt and /config/client/client.key):

spec:
  containers:
    - name: my-container
      image: "my-image"
      imagePullPolicy: Always
      ports:
        - containerPort: 50100
      env:
      - name: CONF_FILE
        value: "/config/srv.yaml"
      volumeMounts:
      - name: yaml-vol
        mountPath: "/config" # KLUDGY
        readOnly: true
      - name: certs-vol
        mountPath: "/config/client"
        readOnly: true

    volumes:
    - name: yaml-vol
      secret:
        secretName: my-yamls # KLUDGY
    - name: certs-vol
      secret:
        secretName: my-certs # contains the *.crt/*.key cert files

However it involves storing the entire /config/srv.yaml in the kubernetes secrets my-yamls.

The k8s secrets docs suggest there's a way to create a dynamic YAML config for one's containers - filling in the secrets precisely where needed, using stringData e.g.

stringData:
  config.yaml: |-
    apiUrl: "https://my.api.com/api/v1"
    username: {{username}}
    password: {{password}}

but the docs trail off with a very vague:

Your deployment tool could then replace the {{username}} and {{password}} template variables before running kubectl apply.

I just need to fill in two string items in a dynamic config: clientid and tenantid.

Using just kubectl, how can one create a dynamic YAML for a container - storing the non-sensitive YAML template in the deploy.yaml - and have just the sensitive items in k8s secrets?

If you're willing to write a simple app (e.g. Python, Go), it would be trivial to pass off this task to the app rather than just using `kubectl` plus shell commands. — Cloud, Nov 27 '19 at 15:57
@Cloud If one could change the app logic, what is the best practices for passing such secrets to it? Remove any "string" secrets from the `app.yaml` and pass them instead via `ENV VAR`? — colm.anseo, Nov 27 '19 at 16:40
I was referring to dynamically creating the `yaml` file with *just* the pieces you needed, by using a Python script to load the `yaml` file (it can be imported as an object, similar to JSON data), extract just the secrets you're interested in, and write those secrets alone to a new `yaml` file, which can be deployed either via `kubectl`, or within Python using the [K8S Py API](https://github.com/kubernetes-client/python). Please avoid using env vars when possible, as they're ugly (i.e. globals), and tend to create security holes. — Cloud, Nov 27 '19 at 16:45
Also, env vars make it easy to poke around in `/proc` and snatch data if the system isn't properly secured. — Cloud, Nov 27 '19 at 16:47
Agreed about `ENV VAR`s. I follow your workaround. But if one were creating a service from scratch whats the best practice for inputing secrets to said service? Have an `app.yaml` for service stuff - and a `secrets.yaml` with all the sensitive strings? — colm.anseo, Nov 27 '19 at 16:49
My off-the-cuff response/approach would be a Helm chart that creates the secrets, plus a deployment/pod definition that requests said secrets by key/name. Helm just released v3, so probably best to go with that versus v2. It's my preferred approach for deployment in general, since it allows users to (easily) override values in a standardized (and automation-friendly) manner. — Cloud, Nov 27 '19 at 17:06

score 3 · Accepted Answer · answered Nov 28 '19 at 07:59

An alternative will be to use another tool for secret management. One solution will be to use Kamus. Kamus support templating so you can do something like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: encrypted-secrets-cm
data:
  tenantid: <encrypted>
  clientid: <encrypted>
  template.ejs: |
     tenantid: <%- secrets["tenantid"] %>
     clientid: <%- secrets["clientid"] %>
     certfile: '/config/client/client.crt'
     keyfile:  '/config/client/client.key'

Where the values are encrypted using Kamus.

And then either use clientSecret and store it the same way, or create a regular secret for both the crt and key. It's worth noticing that (assuming this is Azure) client id and tenant id are not considered secrets, and can be committed to a private repository.

Full disclosure: I'm Kamus author.

generate dynamic runtime YAML for container with k8s secrets

1 Answers1