Kubernetes container secret not recognized by app as env variable when app starts causing it to fail

Question

Im having an issue with deploying my app to my Kubernetes cluster in Digital Ocean and I cannot for the life of me figure out how to solve this issue.

I'm creating a python flask api with Celery using cloudamqp to handle tasks for my api routes. All of this works well but the issue has to do with env variables. In my local machine, everything works well. When I deploy my app to my kubernetes cluster, the containers (one for flask-api and the other for api-worker) wont start due to both containers not being able to read the env variables. Ive used the secrets file approach from the kubernetes docs: kubernetes handle secrets

env.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: backend
type: Opaque
data:
  APP_PORT: <base 64 encoded number>
  JWT_KEY: <base 64 encoded string>
  CLOUDAMQP_URL: <base 64 encoded string>

deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: my-api:latest
          resources:
            requests:
              cpu: "200m"
              memory: "300Mi"
            limits:
              cpu: "300m"
              memory: "600Mi"
          envFrom:
            - secretRef:
                name: my-secret
          ports:
            - containerPort: 5000

        - name: api-worker
          image: api-worker:latest
          resources:
            requests:
              cpu: "200m"
              memory: "300Mi"
            limits:
              cpu: "300m"
              memory: "600Mi"
          envFrom:
            - secretRef:
                name: my-secret

---
apiVersion: v1
kind: Service
metadata:
  name: api-server
  namespace: backend
spec:
  selector:
    app: api-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000

When I deploy to kubernetes using kubectl commands, my containers fail because the code cant recognize the env variable.

ex:

from celery import Celery
import os

def create_celery_app():
    broker_url = os.environ['CLOUDAMQP_URL']
    celery = Celery('tasks', broker=broker_url, backend='rpc://')
    celery.conf.broker_connection_retry_on_startup = True
    return celery

celery = create_celery_app()

The issue is definitely the env variable (in this case CLOUDAMQP_URL. It exists within my secrets file and Ive tested to see if the values show up. When I hardcode the values into the code above, the app works perfectly fine when deployed to my cluster. I think the issue is related to the app starting before the env variables are set (but I could be mistaken). I was able to print an env variable on an api route but that is after the app was fully operational and running. The app seems to fail on startup so for example, if the function above was in a main.py file and the app is starting, it crashes. Anyone have a solution for this or faced something similar? I checked the shells of the container (when the app isnt crashing) and I can see the env variables with the values I put in the env.yaml file in there and they are accurate.

Answer 1

So one way I got it to work was to use configMap instead of secrets. I will still try to get secrets to work but for now, this is how I got it work. I wanted to create an automated process for this so I may create a script that reads my env variables and overwrites my existing configMap with the new values or deletes the old configMap and replaces with a new one (I have to test which works).

Using kubectl, I created a configMap in my namespace called backend.

kubectl create configmap my-config-map \
  --from-literal=APP_PORT=<my_port>\
  --from-literal=JWT_KEY=<my_jwt_key> \
  --from-literal=CLOUDAMQP_URL=<my_url> \
  -n backend

One thing to note here, if you are storing numbers, you will have to convert them to ints or floats when you are reading the environment variable in your app like so:

import os

app_port = int(os.environ['APP_PORT'])

I also changed my deployment.yaml file. For completion sake, I provided the full deployment file if anyone needs help with the structure.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api-server
          image: my-api:latest
          envFrom:
            - configMapRef:
                name: my-config-map
          resources:
            requests:
              cpu: "200m"
              memory: "300Mi"
            limits:
              cpu: "300m"
              memory: "600Mi"
          ports:
            - containerPort: 5000

        - name: api-worker
          image: api-worker:latest
          envFrom:
            - configMapRef:
                name: my-config-map
          resources:
            requests:
              cpu: "200m"
              memory: "300Mi"
            limits:
              cpu: "300m"
              memory: "600Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: api-server
  namespace: backend
spec:
  selector:
    app: api-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000