13

I want to learn how to update secrets in worker pods without killing and recreating the deployment.

Currently pods pull in secrets to as env vars with:

        env:
        - name: SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: secret_access_key
              name: secrets

but this only happens when they startup.

So if there is a need to change a secret I have to:

  1. Change the secret in secrets.yaml
  2. kubectl apply -f secrets.yaml
  3. kubectl delete -f worker-deployment.yaml
  4. kubectl apply -f worker-deployment.yaml

I really don't like step 3 and 4 as they terminate jobs in progress.

What is a better workflow for updating env var secrets in place?

ProGirlXOXO
  • 2,170
  • 6
  • 25
  • 47

3 Answers3

15

There is no way to do a "hot reload" for pod's environment variables.

Although, you do not need to delete and recreate the deployment again to apply the new secret value. You only need to recreate the underlying pods. Some options are:

rollout restart is only available on kubernetes v1.15+

Eduardo Baitello
  • 10,469
  • 7
  • 46
  • 74
  • 3
    You could also add an annotation containing the checksum of the configmap, or the timestamp of its last modification. Slighly cleaner than abusing an unrelated filed in the template. – Paulo Schreiner Apr 13 '19 at 12:34
  • 1
    @PauloSchreiner on k8s v1.15 the `kubectl rollout restart` is a better option now to trigger the rolling update. – Eduardo Baitello Feb 19 '20 at 12:20
5

As already mentioned, what you want to do is not possible. However, there is an alternative offered by Kubernetes: mounting ConfigMaps as Volumes. For example

apiVersion: v1
kind: Pod
metadata:
  name: configmap-pod
spec:
  containers:
    - name: test
      image: busybox
      volumeMounts:
        - name: config-vol
          mountPath: /etc/config
  volumes:
    - name: config-vol
      configMap:
        name: log-config
        items:
          - key: log_level
            path: log_level

In this case, the log-config ConfigMap would be mounted as a Volume, and you could access the contents from its log_level entry as the file “/etc/config/log_level” inside the pod.

Changes to the config map are reflected by changes in the files on the Volume, and those changes can, in turn, be watched by your application by using the appropriate functionality in your language.

Paulo Schreiner
  • 1,046
  • 8
  • 9
-6

You are able to update secrets via a tool like K8S's WebUI (Dashboard) or fabric8 if you have the appropriate permissions.

Check out Kubernetes modify a secret using the cli for other options.

dejon97
  • 299
  • 2
  • 4
  • 1
    The question is not about how to modify secrets (it's already done in step 1 and 2). It's about handling the reload of pod configs. – Eduardo Baitello Apr 12 '19 at 00:58
  • Curious why you thought I didn't understand the question. FWIW, I have personally used the Dashboard and Fabric8 to change the value of a secret which updated the corresponding environment value in the pod. If you are unaware of those options, I would be happy to demonstrate them for you. – dejon97 Apr 12 '19 at 16:20
  • 2
    Updating the corresponding environment without recreating the pods? Would be awesome to see this working! (Especially using the kubernetes dashboard). Can you create a POC to demonstrate it? Consider including the demonstration in [#24957](https://github.com/kubernetes/kubernetes/issues/24957) and [#22368](https://github.com/kubernetes/kubernetes/issues/22368), the people of Kubernetes project are unaware of this feature too. – Eduardo Baitello Apr 12 '19 at 16:59
  • I stand corrected. You can modify the secret values via the Dashboard and Fabric8; however, you must restart the pod(s) for any changes to be reflected in the environment. I appreciate the info Educardo. – dejon97 Apr 12 '19 at 21:12