64

I am trying to create a Helm Chart with the following resources:

  1. Secret
  2. ConfigMap
  3. Service
  4. Job
  5. Deployment

These are also in the order that I would like them to be deployed. I have put a hook in the Deployment so that it is post-install, but then Helm does not see it as a resource and I have to manually manage it.

The Job needs the information in the Secret and ConfigMap, otherwise I would make that a pre-install hook. But I can't make everything a hook or nothing will be managed in my release.

Does anyone have a solution or idea to be able to manage all of the resources within the Helm release AND make sure the Job finishes before the Deployment begins?

My only thought right now is two make two Charts: One with 1-4 and the second with 5 which would depend on the first.

waterprincess
  • 675
  • 1
  • 5
  • 6
  • Have you already tried using [a multi-document YAML file](https://github.com/helm/helm/blob/v2.10.0/docs/chart_template_guide/yaml_techniques.md#embedding-multiple-documents-in-one-file)? Multi-document YAML is an ordered list, but I don't know if they are _applied_ in that order. I would *hope so* but I hope lots of things – mdaniel Aug 22 '18 at 00:57
  • Out of interest what does the Job do? – Ryan Dawson Aug 22 '18 at 07:42
  • I ask partly because I'm wondering if you could put an initContainer in the Deployment and move the Job's logic there instead. The configmap and secret should be created first anyway with helm's resource ordering – Ryan Dawson Aug 22 '18 at 07:46
  • @RyanDawson The Job prepares the Database and I wouldn't want it running for each Deployment Pod. – waterprincess Aug 22 '18 at 13:57
  • 2
    If you choose to do so you could make it idempotent with a check at the beginning. For examples of doing this from the official helm/charts repo see any of https://github.com/helm/charts/search?q=%22init-data%22&unscoped_q=%22init-data%22 or https://github.com/helm/charts/blob/master/stable/keycloak/templates/statefulset.yaml#L33 – Ryan Dawson Aug 22 '18 at 15:20

2 Answers2

133

Helm collects all of the resources in a given Chart and it's dependencies, groups them by resource type, and then installs them in the following order (see here - Helm 2.10):

  1. Namespace
  2. ResourceQuota
  3. LimitRange
  4. PodSecurityPolicy
  5. Secret
  6. ConfigMap
  7. StorageClass
  8. PersistentVolume
  9. PersistentVolumeClaim
  10. ServiceAccount
  11. CustomResourceDefinition
  12. ClusterRole
  13. ClusterRoleBinding
  14. Role
  15. RoleBinding
  16. Service
  17. DaemonSet
  18. Pod
  19. ReplicationController
  20. ReplicaSet
  21. Deployment
  22. StatefulSet
  23. Job
  24. CronJob
  25. Ingress
  26. APIService

During uninstallation of a release, the order is reversed (see here).

Following this logic, in your case when your Job resource is created, both the Secret and the ConfigMap will already be applied, but Helm won't wait for the Job to complete before applying the Deployment. If you split your Chart to two parts (1-4, 5) and install them sequentially you would still have the problem of the Deployment being possibly applied before the Job is completed. What I would suggest is splitting your Chart to two parts (1-3, 4-5), in which the the Job has a pre-install hook, which would make sure it completes before your Deployment is applied.

Yaniv Oliver
  • 3,372
  • 1
  • 19
  • 20
  • thank you for your suggestion, I will read more about Chart dependencies and try out splitting it into (1-3,4-5). My follow-up question is then, is this a 'best practice' or am I already straying from that ideal with my problem to begin with? Also, thank you for the links to the code, it is hard for me to find helpful things like that! – waterprincess Aug 22 '18 at 14:03
  • 1
    I'm not sure what the best practice here is, but as far as I see it there are two options - splitting your chart or using init containers to make your k8s resource idempotent. There are pros and cons for both (mainly having simpler k8s resources with less complex logic when choosing to split your charts, but requires more complex logic when applying your Helm charts and managing releases). IMHO I'd go with splitting the charts in your case. – Yaniv Oliver Aug 23 '18 at 10:25
  • Nice answer and a simple explaination – Venkata S S K M Chaitanya Mar 26 '19 at 15:05
  • 17
    Link to source for v3 of helm:https://github.com/helm/helm/blob/release-3.0/pkg/releaseutil/kind_sorter.go – object88 Jan 28 '20 at 20:17
  • what's the order of type `List`? – Lei Yang Mar 18 '20 at 06:13
  • And if I haver another kind not in that list like SecretProviderClass? – Alexsandro Oct 03 '20 at 15:08
  • 1
    @Alexsandro I had the same question in my mind as well and it seems to be answered here: https://github.com/helm/helm/blob/ac925eb7279f4a6955df663a0128044a8a6b7593/pkg/releaseutil/kind_sorter.go#L139 – Raymond Tau May 13 '21 at 01:09
  • What about custom resources? Facing an issue with OPA templates and constraints not installing in the right order. – Marco Aug 16 '21 at 09:18
  • what's the order of `HorizontalPodAutoscaler`? – Lei Yang Aug 31 '21 at 07:13
9

Helm tries to install things in a certain order, but doesn't check if pods / deployments / jobs are running / completed before moving on. Also note that a chart and its dependencies are installed simultaneously, so you cannot use a chart with a dependency to re-order how Helm installs resources.

You can use chart hooks to change the order, but these aren't managed resources. In my case, the problem was that we needed custom resources up, then we needed a short script to run, and then we needed to start our deployments. With --wait, if the pod the script was in completed, Helm would mark the upgrade/install as a failure and rollback. The solution in this case was just to use a Job instead of a Pod, which commenters on that issue had more problems with than I did, and then accept that the deployments would restart a few times before everything finally became ready.

Helm 3.7 install order:

  1. Namespace
  2. NetworkPolicy
  3. ResourceQuota
  4. LimitRange
  5. PodSecurityPolicy
  6. PodDisruptionBudget
  7. ServiceAccount
  8. Secret
  9. SecretList
  10. ConfigMap
  11. StorageClass
  12. PersistentVolume
  13. PersistentVolumeClaim
  14. CustomResourceDefinition
  15. ClusterRole
  16. ClusterRoleList
  17. ClusterRoleBinding
  18. ClusterRoleBindingList
  19. Role
  20. RoleList
  21. RoleBinding
  22. RoleBindingList
  23. Service
  24. DaemonSet
  25. Pod
  26. ReplicationController
  27. ReplicaSet
  28. Deployment
  29. HorizontalPodAutoscaler
  30. StatefulSet
  31. Job
  32. CronJob
  33. Ingress
  34. APIService
  35. this closed git issue tells us CustomResources are last to be installed.

Source. Only difference from above is more resource types and the ServiceAccount got pushed up slightly in the list.

jeremysprofile
  • 10,028
  • 4
  • 33
  • 53