3

We are using helm charts to deploy our charts in Openshift.

This is our workflow:

  • We create a new version of the helm and docker image at each sprint/ e.g 1.0.0 (saving them in a snapshot artifactory)
  • During the sprint we build several times the helm chart and the docker image and push them in our snapshot artifactory every time.
  • Once the helm chart and the docker image are published, we automatically deploy our chart in our test environment
  • once we are ready we create the production version of the charts and the docker image: we basically publish the helm chart and the docker image in a release artifactory with the same version. From now on the helm chart and the docker images are immutable
  • Now we deploy in PROD

The issue is that usually the helm-chart does not change but the docker image v1.0.0 (snapshot) may change several times during the sprint therefore when we try to upgrade the helm chart in our test env, helm does not detect any change and then the application is not updated.

To solve this situation, currently, every time that we have to deploy in the test environment, we uninstall the application and re install the helm chart (with the image pull policy == always)

I was wondering if there is a way to modify our helm chart in order to force it to redeploy when we build a new version. e.g we tried to add an annotation in the deployment.yaml : build-time: {{ now }} but this changes every time so the helm chart is always redeployed in the test environment (and usually is fine but not if we trigger a manual re-deploy of all our components).

Is it possible for example to provide a parameter during the helm package command? Something like helm package --set package-time=timestamp and then we could save this value as annotation.

Any better solution?

Fabry
  • 1,498
  • 4
  • 23
  • 47
  • 1
    In general, you should not reuse docker tags because of problems like this. Append something unique to each docker tag (git-sha of the commit used to build, epoch timestamp, etc) and then just change the tag with helm `helm update --set image.tag=mynewtag` – jordanm Aug 10 '22 at 19:06
  • Some systems like [fluxcd](https://fluxcd.io/) can automatically scan your repo for new images and update your helm settings for new deployments. – jordanm Aug 10 '22 at 19:07

2 Answers2

2

In addition to you functional tag (eg v1.0.0), add a second tag to your image with something unique, for example the git-sha commit tag coming from git used at build time.

The functionnal tag is a "floating" tag, it will move from one image to the other but the "git-sha" tag will be unique

One way to get a short version of git-sha:

git log -n 1 --pretty=format:'%h'

In your deployment, specify the "git-sha" tag for your image in your helm chart as a variable

The functional tag/version could be read from a single line file in your source, So when you are ready to push v2.0.0, just change the file and commit it

titou10
  • 2,814
  • 1
  • 19
  • 42
  • I have to check if from the pipeline I can retrieve the previous git-sha used to tag the snapshot image in order to re-tag the image for the release artifactory – Fabry Aug 11 '22 at 06:27
  • @Fabry, the git command is: "git log -n 1 --pretty=format:'%h'" to retrieve the "short" version of the current sha commit – titou10 Aug 11 '22 at 15:22
0

There is two options to solve the problem:

  1. Use an incremental tag in docker image because using the same tag for a different image is not a good especially if you have multiple services which use the image
  2. The solution you are searching: force the helm chart to roll the deployment (or statefulset) on each helm upgrade. To achieve that, you can add a new annotation to your deployment template, with a random number as a value, here are the instructions from the official documentation:
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}
Hussein Awala
  • 4,285
  • 2
  • 9
  • 23
  • for #1 I have to check if from the pipeline I can retrieve the previous git-sha used to tag the snapshot image in order to re-tag the image for the release artifactory fro #2 this will always re-deploy even if the image is not changed at all – Fabry Aug 11 '22 at 06:27
  • As I understood, you have a CD pipeline which call `helm upgrade` if there is a change on the helm chart and/or the docker image, so there is no problem if you roll the deployment each time... Especially as you can configure your deployment to avoid downtime (rolling updates) – Hussein Awala Aug 11 '22 at 07:21
  • The answer that @HusseinAwala gave you is the correct and best approach. As the underlying image is changing and yet the image tag is staying the same, then you have no knowledge of when to roll and when to not roll a new deployment. We have also faced the problem of changing configmaps, and so if you are using configmaps as well, please make sure to create a hash of the config map in deployment annotation, so after the configmap changes, you can roll a new deployment as well. Having different software under the same tag is not a best practice. It poses a lot of problems. – Pavol Krajkovič Aug 11 '22 at 15:40