16

Just to clarify.

  1. I am not talking about Helm chart best practices. I can read them by myself over here: https://helm.sh/docs/chart_best_practices/
  2. I am not talking about Terratest.

I am interested in Helm Test you can read more about it over here https://helm.sh/docs/chart_tests/#chart-tests .

Unfortunately, the documentation doesn't provide you a lot of information and that is partly the reason for this question.

What we have:

  1. We can have a test defined in a Chart by the following path

our-app/templates/tests/test-our-app.yaml

  1. As far as I can understand and according to documentation, it is not a unit test for chart but more like a smoke test or integration test. Documentation: "validate that your chart works as expected when it is installed". This is exactly what I need.
  2. In order to run the test https://helm.sh/docs/chart_tests/#steps-to-run-a-test-suite-on-a-release. We need to install our-app Chart
    $ helm install our-app

and then we can test that image

    $ helm test release-with-our-app

Things I would like to clarify:

  1. Should we have two separate docker images or one for application and tests?...

    a). In the case of it being single one image for both application code and test code then we are adding code for the test/tests all on that image, which makes it bigger and additionally we need to put dependencies that are required by tests on that image. To me, this solution seems to be wrong.

    b). The case where we have a separate image for tests makes more sense because our app docker image is free of any unnecessary dependencies. What's more, in the second solution we don't really care about the size of the docker image with the test(s) because it is supposed to be a short-living image that shuts down when tests on it are finished.

    Is my assumption correct and we should have a separate image for these tests?

  2. In case if have two separate images, one for code another for tests. How do we marry them with each other? Does it mean that we have to pass a Build/Release number to helm test command so it is able to pull down the correct image for tests?

  3. Also in case if we have two separate images. Should I specify the test run in a docker file for tests? for example :

   CMD ["sh", "-c", "gradle test -Denvironment=$ENVIRONMENT"]

Or should it be inside chart

   our-app/templates/tests/test-our-app.yaml

For example test.yaml from mysql repo:

https://github.com/helm/charts/blob/master/stable/mysql/templates/tests/test-configmap.yaml

  1. It looks like environments are not mentioned at all in the documentation. My question is what is the best/recommended way to pass environment name to the chart so tests know which endpoints to hit etc? Should it be Chart args How to pass dynamic arguments to a helm chart that runs a job ?
Sergii Zhuravskyi
  • 4,006
  • 5
  • 18
  • 26

2 Answers2

16

an example you can find in mysql chart by the following link https://github.com/helm/charts/tree/master/stable/mysql/templates

Answering my own question:

  1. We made a separate docker image for helm tests due to the following reasons:
    1.1 When helm smoke docker image is separate that makes it free of any constraints of the main app docker image. So you can install any libraries and avoid following some security guidelines that main app has to follow.
    1.2 There is no restriction on the size of a docker image for helm test becasue it is short living image , the image of helm test is a live for a duration of tests inside.
  2. We marry them via Chart version. So the image of main application has always the same version as the helm tests docker image. enter image description here

We don't have to pass anything, when you run "helm test yourapp", helm looks for successfully deployed version of yourapp and runs helm test with that version.
enter image description here
3. When you run the "helm test yourapp" command. Helm looks in file
yourapp/templates/tests/smoke-test.yaml
calls command that is specified by the following path:
spec/containers/command: ["sh", "-c", "your command"] plz have a look on the screenshot or go to git repo of mysql chart https://github.com/helm/charts/tree/master/stable/mysql/templates enter image description here

  1. The best way to pass data and Env specific data is passing them via config maps, env variables, and secrets as Hanx unanswered. If it is stored in such a way then you can populate such data via helm values files and it will be easy to update. We found that having env specific data inside image is a bad solution because if for example a URL/host changes in some environment we need to create a new build instead of just updating a values file and performing deploy.
Sergii Zhuravskyi
  • 4,006
  • 5
  • 18
  • 26
  • It is a very informative answer. Could you elaborate on how do you read test results? Are they only available as logs from test pod or you have found a better way? – jozala Oct 28 '20 at 23:00
  • in our case, we don't need test specific logs, if there is a problem with this type of test we look into the application logs. As far as we understood helm test should be a small/tiny set of tests that verify that the application started, for example, we verify that endpoints are responding correctly 200 or something. – Sergii Zhuravskyi Oct 29 '20 at 09:16
  • Helm tests are not substitution of Functional Tests that has to be running in your CI/CD pipeline. They are not substitution of Smoke functional testing in your CI/CD pipeline either. In our understanding, it is something that Devops /Software engineers run just to see that a pod started after deployment and there are no critical mistakes in a config(values files etc). – Sergii Zhuravskyi Oct 29 '20 at 09:16
  • I don't think it would be smart to make helm tests big or complicated for obvious reasons, it is difficult to debug them, additionally, you want result quick so you can proceed with deployment of other applications and then in the end run your functional tests. – Sergii Zhuravskyi Oct 29 '20 at 09:16
  • Thanks, for your comment. I was looking at using Helm tests for smoke tests (functional). I got the impression it is a good idea after reading FluxCD and Flagger documentation. It is not stating that directly though. We are going to use FluxCD so deployment part of our pipeline is inside Kubernetes anyway. We are currently running smoke tests from CI and never had a need to debug them. I will try to look more into why it may be a bad idea. Thank you for valuable input anyway. – jozala Oct 29 '20 at 21:19
  • For a public helm chart, I agree: the chart test should just test that the chart is correct; it doesn't know the context of its deployment so it can't really do more than that. But for private charts that are used as part of CI/CD, I don't see anything wrong with using helm test. It nicely couples the test to what is actually deployed in test environment, and it makes it easy to troubleshoot in other deployments such as staging, or a debugging environment setup specifically to replicate an issue foundin prod. – Oliver Jun 04 '21 at 16:02
3

Just starting this topic:

In my opinion it depends on your needs.

For the 1,2,3 - as I understood "helm test" is very flexible and it can perform any additional tests depends on your needs.

You can use different images and different approaches or one image with different arguments but sometimes probably it can be impossible due to docker entrypoint - please see commands and arguments.

According to the 4 question - in my opinion you should consider using configmaps and secrets to define and expose env variables into your pods.

Additional resources "helm test":

Hope this help.

Mark
  • 3,644
  • 6
  • 23