How to properly configure the environment in kubernetes cluster?

Question

I have a spring boot application with two profiles, dev and prod, my docker file is:

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-Dspring.profiles.active=dev","-cp","app:app/lib/*","com.my.Application"]

please not that, when building the image, I specify the entrypoint as command line argument.

This is the containers section of my kubernetes deployment where I use this image:

containers:
  - name: myapp
    image: myregistry.azurecr.io/myapp:0.1.7
    imagePullPolicy: "Always"
    ports:
    - containerPort: 8080
      name: myapp
    readinessProbe:
      httpGet:
        path: /actuator/health
        port: 8080
      timeoutSeconds: 3
      periodSeconds: 20
      failureThreshold: 3

It works but has a major flaw: how can I now switch to the production environment without rebuilding the image?

The best would be to remove that ENTRYPOINT in my docker file and give this configuration in my kubernetes yml so that I could always use the same image...is this possible?

edit: I saw that there is a lifecycle istruction but note that I have a readiness probe based on the spring boot's actuator. It would always fail if I used this construct.

score 3 · Accepted Answer · answered Jul 21 '19 at 09:35

You can override an image's ENTRYPOINT by using the command property of a Kubernetes Pod spec. Likewise, you could override CMD by using the args property (also see the documentation):

containers:
- name: myapp
  image: myregistry.azurecr.io/myapp:0.1.7
  imagePullPolicy: "Always"
  command: ["java","-Dspring.profiles.active=prod","-cp","app:app/lib/*","com.my.Application"]
  ports:
  - containerPort: 8080
    name: myapp

Alternatively, to provide a higher level of abstraction, you might write your own entrypoint script that reads the application profile from an environment variable:

#!/bin/sh

PROFILE="${APPLICATION_CONTEXT:-dev}"
exec java "-Dspring.profiles.active=$PROFILE" -cp app:app/lib/* com.my.Application

Then, you could simply pass that environment variable into your pod:

containers:
- name: myapp
  image: myregistry.azurecr.io/myapp:0.1.7
  imagePullPolicy: "Always"
  env:
  - name: APPLICATION_CONTEXT
    value: prod
  ports:
  - containerPort: 8080
    name: myapp

score 3 · Answer 2 · answered Jul 21 '19 at 09:56

Rather than putting spring.profiles.active in dockerfile in the entrypoint.

Make use of configmaps and application.properties.

Your ENTRYPOINT in dockerfile should look like:

ENTRYPOINT ["java","-cp","app:app/lib/*","com.my.Application","--spring.config.additional-location=/config/application-dev.properties"]

Create a configmap that acts as application.properties for your springboot application

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
  namespace: flow
data:
  application-dev.properties: |
    spring.application.name=myapp
    server.port=8080
    spring.profiles.active=dev

NOTE: Here we have specified spring.profiles.active.

In containers section of my kubernetes deployment mount the configmap inside container that will act as application.properties.

containers:
  - name: myapp
    image: myregistry.azurecr.io/myapp:0.1.7
    imagePullPolicy: "Always"
    command: ["java","-cp","app:app/lib/*","com.my.Application","--spring.config.additional-location=/config/application-dev.properties"]
    ports:
    - containerPort: 8080
      name: myapp
    volumeMounts:
    - name: myapp-application-config
      mountPath: "/config"
      readOnly: true
    volumes:
    - name: myapp-application-config
      configMap:
        name: myapp-config
        items:
        - key: application-dev.properties
          path: application-dev.properties
    readinessProbe:
      httpGet:
        path: /actuator/health
        port: 8080
      timeoutSeconds: 3
      periodSeconds: 20
      failureThreshold: 3

NOTE: --spring.config.additional-location points to location of application.properties that we created in configmaps.

So making use of configmaps and application.properties one can override any configuration of your application wihtout rebuilding the image.

If you want to add a new config or update value of existing config, just make appropriate changes in configmap and kubectl apply it. Then scale down and scale up your application pod, to bring the new config in action.

Hope this helps.

score 1 · Answer 3 · answered Jul 21 '19 at 10:40

There are many many ways to set Spring configuration values. With some rules, you can use ordinary environment variables to specify individual property values. You might see if you can use this instead of having a separate Spring profile control.

Using environment variables has two advantages here: it means you (or your DevOps team) can change deploy-time settings without recompiling the application; and if you're using a deployment manager like Helm where some details like hostnames are intrinsically unpredictable, this lets you specify values that can't be known until deploy time.

For example, let's say you have a Redis dependency:

cache:
  redis:
    url: redis://localhost:6379/0

You could override this at deploy time by setting

containers:
  - name: myapp
    env:
      - name: CACHE_REDIS_URL
        value: "redis://myapp-redis.default.svc.cluster.local:6379/0"

score 0 · Answer 4 · answered Feb 14 '21 at 17:05

One way to do this is using spring cloud Kubernetes as described here https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/index.html#configmap-propertysource

You can define your profiles in a configmap like below

kind: ConfigMap
apiVersion: v1
metadata:
  name: demo
data:
  application.yml: |-
    greeting:
      message: Say Hello to the World
    farewell:
      message: Say Goodbye
    ---
    spring:
      profiles: development
    greeting:
      message: Say Hello to the Developers
    farewell:
      message: Say Goodbye to the Developers
    ---
    spring:
      profiles: production
    greeting:
      message: Say Hello to the Ops

And can then select the desired profile by passing an environment variable in your Kubernetes deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-name
  labels:
    app: deployment-name
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deployment-name
  template:
    metadata:
      labels:
        app: deployment-name
    spec:
        containers:
        - name: container-name
          image: your-image
          env:
          - name: SPRING_PROFILES_ACTIVE
            value: "development"

How to properly configure the environment in kubernetes cluster?

4 Answers4