4

I've been using terraform for a while and I really like it. I also set up Atlantis so that my team could have a "GitOps" flow. This is my current process:

  • Add or remove resources from Terraform files
  • Push changes to GitHub and create a pull request
  • Atlantis picks up changes and creates a terraform plan
  • When the PR is approved, Atlantis applies the changes

I recently found myself needing to set up a few managed Kubernetes clusters using Amazon EKS. While Terraform is capable of creating most of the basic infrastructure, it falls short when setting up some of the k8s resources (no support for gateways or ingress, no support for alpha/beta features, etc). So instead I've been relying on a manual approach using kubectl:

  • Add the resource to an existing file or create a new file
  • Add a line to a makefile that runs the appropriate command (kubectl apply or create) on the new file
  • If I'm using a helm chart, add a line with helm template and then kubectl apply (I didn't really like using tiller, and helm3 is getting rid of it anyway)
  • If I want to delete a resource, I do it manually with kubectl delete

This process feels nowhere near as clean as what we're doing in Terraform. There are several key problems:

  • There's no real dry-run. Using kubectl --dry-run or kubectl diff doesn't really work, it's only a client-side diff. Server-side diff functions are currently in alpha
  • There's no state file. If I delete stuff from the manifests, I have to remember to also delete it from the cluster manually.
  • No clear way to achieve gitops. I've looked at Weaveworks Flux but that seems to be geared more towards deploying applications.
  • The makefile is getting more and more complicated. It doesn't feel like this is scaleable.

I should acknowledge that I'm fairly new to Kubernetes, so might be overlooking something obvious.

Is there a way for me to achieve a process similar to what I have in Terraform, within the Kubernetes universe?

kenske
  • 2,235
  • 2
  • 20
  • 27
  • 1
    I personally agree with your description of the problem here as it mirrors experiences I have had for the last year myself. If you want something that cleanly chains your Terraform cluster creation into your Helm deployments, but provides a more robust solution than raw `kubectl`, may I recommend the `k8s` module within Ansible? It can bridge the gap between Terraform's infra setup and Helm's deployments quite well. Make sure the `kubeconfig` output by Terraform is picked up by Ansible. I chain all this stuff together myself in a Jenkins Pipeline, but your choice should work well too. – Matthew Schuchard Apr 08 '19 at 15:58

1 Answers1

1

This is more of an opinion question so I'll answer with an opinion. If you like to manage configuration you can try some of these tools:

  • If you want to use existing YAML files (configurations) and use something at a higher level you can try kustomize.
  • If you want to manage Kubernetes configurations using Jsonnet you should take a look at Ksonnet. Keep in mind that Ksonnet will not be supported in the future.

If you want to just automatically do a helm update in an automated way, there is not a tool there yet. You will have to build something at this point to orchestrate everything. For example, we ended up creating an in house tool that does this.

Rico
  • 58,485
  • 12
  • 111
  • 141
  • Ksonnet is no longer supported. "The team behind ksonnet is stepping back from the project. As a result, work on ksonnet will end and the GitHub repositories will be archived". https://ksonnet.io/ – kenske Apr 08 '19 at 16:06
  • My issue is not related to templating (Helm has worked fine for me). The question refers to actually applying the changes in an automated way. – kenske Apr 08 '19 at 16:09
  • Gotcha, there isn't really a tool, we ended up creating our own tool that runs automatic commands like `helm upgrade`. – Rico Apr 08 '19 at 16:49