Setting up on ISTIO on EKS cluster using Terraform or Helm

Question

I'm new to Terraform and Helm world! I need to set up Istio on the AWS EKS cluster. I was able to set up the EKS cluster using Terraform. I'm thinking of installing ISTIO on top of the EKS cluster using Terraform by writing terraform modules. However, I found that we can set up Istio on top of eks using the helm chart.

Can someone help me to answer my few queries:

1. Should I install Istio using Terraform? If yes, Is there any terraform module available or How can I write one?
2. Should I install Istio using Helm Chart? If yes, what are the pros and cons of it?
3. I need to write a pipeline to install Istio on EKS cluster. Should I use a combination of both terraform and Helm as provider?

Thank you very much for your time. Appreciate all your help!

Answer 1

To extend @Chris 3rd option of terraform + helm provider,

as for version 1.12.0+ of istio they officially have a working helm repo:

istio helm install

and that with terraform's helm provider Terraform helm provider allows an easy setup that is configured only by terraform:

provider "helm" {
  kubernetes {
    // enter the relevant authentication
  }
}

locals {
  istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}

resource "helm_release" "istio-base" {
  repository       = local.istio_charts_url
  chart            = "base"
  name             = "istio-base"
  namespace        = var.istio-namespace
  version          = "1.12.1"
  create_namespace = true
}

resource "helm_release" "istiod" {
  repository       = local.istio_charts_url
  chart            = "istiod"
  name             = "istiod"
  namespace        = var.istio-namespace
  create_namespace = true
  version          = "1.12.1"
  depends_on       = [helm_release.istio-base]
}

resource "kubernetes_namespace" "istio-ingress" {
  metadata {
    labels = {
      istio-injection = "enabled"
    }

    name = "istio-ingress"
  }
}

resource "helm_release" "istio-ingress" {
  repository = local.istio_charts_url
  chart      = "gateway"
  name       = "istio-ingress"
  namespace  = kubernetes_namespace.istio-ingress-label.id
  version    = "1.12.1"
  depends_on = [helm_release.istiod]
}

This is the last step that was missing to make this production ready

It is no longer needed to locally keep the helm charts with the null_resource

If you wish to override the default helm values it is nicely shown in here: Artifact hub, choose the relevant chart and see the values

Answer 2

As @Matt Schuchard mentioned, this is a bit opinion based question, that's why I will answer that based on my understanding.

---

Question number 1.

1. To answer your question, Should I install Istio using Terraform?, yes, if you follow Devops practices then you should write everything in a code, so I would recommend to do that.

2. As per the second part of your question, If yes, Is there any Terraform module available, no, from what I see currently there is no Istio module for Terraform, there is only a helm one.

3. As for the last part of the first question, How can I write Terraform module? I would recommend to start with the Terraform documentation. There is also a tutorial for creating a module.

---

Question number 2.

1. To answer your question, Should I install Istio using Helm Chart?, depends on your use case, you can do it either with helm or istioctl/istio operator.
2. As for the following question, If yes, what are the pros and cons of it? I'm not sure if the current helm chart is production ready, according to Istio documentation, Providing the full configuration in an IstioOperator CR is considered an Istio best practice for production environments, so from what I understand, you should rather use operator than helm. Also worth to note that the helm chart was not used by several versions, if was broughts back to life in version 1.8.

---

Question number 3.

1. As per the last question, I need to write a pipeline to install Istio on EKS cluster. Should I use a combination of both terraform and Helm as provider?, depends, it could be either Terraform and Helm, but from what I see it's also possible to do that with an Terraform and Istio Operator, there is an example. So it's rather up to you to decide which path will you take.

---

I would also recommend to take a look at this reddit thread. You might find few useful comments from the prod environment here, about installing Istio with Terraform.

Answer 3

I have been researching this in the last months and want to add my findings to @Jakob's answer:

First, there is an answer to the pros/cons of the different installation method, so I will not say anything about that: https://istio.io/latest/faq/setup/#install-method-selection Basically all of them can be done with terraform in a certain way.

1. terraform + istioctl with terraform null_resource provider

This is basically the istioctl install -f <file> command. You can create a template file and to the istictl install command with the null_resource provider.

resource "local_file" "setup_istio_config" {
  content = templatefile("${path.module}/istio-operator.tmpl", {
    enableHoldAppUntilProxyStarts = var.hold_app_until_proxy_starts
  })
  filename = "istio-operator.yaml"
}

resource "null_resource" "install_istio" {
  provisioner "local-exec" {
    command = "istioctl install -f \"istio-operator.yaml\" --kubeconfig ../${var.kubeconfig}"
  }
  depends_on = [local_file.setup_istio_config]
}

Pros:

• Very easy setup

Cons:

• How to upgrade using istioctl upgrade -f <file has to be solved somehow
• istioctl must be installed in different versions when handling multiple clusters with different istio versions
• Right istioctl version must be choosen on setup

I guess you can solve the upgrade process somehow, but the hole process is not really "infrastructure as code" enough. I didn't look into it further, because it doesn't seam to be good practice.

2. terraform + istio operator with terraform null_resource provider and kubectl provider

Similar the istio operator setup initializes the operator pod and takes a istio-operator.yml to setup istio for you.

resource "null_resource" "init_operator" {
  provisioner "local-exec" {
    command = "istioctl operator init --kubeconfig ../${var.kubeconfig}"
  }
}

resource "kubectl_manifest" "setup_istio" {
  yaml_body = <<YAML
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
    name: istio-setup
    namespace: istio-system
spec:
  profile: default
  hub: gcr.io/istio-release
  tag: 1.9.2
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
  meshConfig:
    defaultConfig:
      holdApplicationUntilProxyStarts: ${var.hold_app_until_proxy_starts}"
YAML
  depends_on = [null_resource.init_operator]
}

It would be a good idea to wait for some seconds between the init and applying the config.

Here is a good article about doing this with Azure's aks: https://medium.com/@vipinagarwal18/install-istio-on-azure-kubernetes-cluster-using-terraform-214f6d3f611

Pros:

• Easy to setup
• Easy to upgrade istio using the kubectl provider

As long as helm is in alpha, this might be the best approach.

3. terraform + helm with terraform helm provider

Istio provides some charts for the different componentes, when downloading istioctl. Those can be used for installing it with helm.

resource "helm_release" "istio_base" {
  name       = "istio-base"
  chart      = "./manifests/charts/base"
  namespace  = "istio-system"
}

Cons:

• Not ready for production

Bonus

4. istio manifest + helm

Some time ago I've read an article on how to use istio manifest from istioctl manifest generate in combination with helm to install and mange istio. This approach needs some custom code, but it could be done with terraform and the helm provider as well.

Please read: https://karlstoney.com/2021/03/04/ci-for-istio-mesh/index.html

Conclusion

Installing istio with terraform works but seams to be a bit dirty at the moment. Once the helm setup is stable, I guess this would be the best approach. And with the helm provider it can be composed with terraform creation of other resources. Terraform certainly misses an istio provider, but I don't think they will create one in the foreseeable future.

Answer 4

For all those who found @Benda's solution to the point. Here is the working template for the same. Since I faced a couple of issues with that template, I compiled it for my own use case. I hope its helpful.

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}
    
provider "kubernetes" {
  config_path    = "~/.kube/config"
}
  
locals {
  istio_charts_url = "https://istio-release.storage.googleapis.com/charts"
}
    
resource "kubernetes_namespace" "istio_system" {
  metadata {
    name = "istio-system"
    labels = {
      istio-injection = "enabled"
    }
  }
}
    
resource "helm_release" "istio-base" {
  repository       = local.istio_charts_url
  chart            = "base"
  name             = "istio-base"
  namespace        = kubernetes_namespace.istio_system.metadata.0.name
  version          = ">= 1.12.1"
  timeout          = 120
  cleanup_on_fail  = true
  force_update     = false
}
    
resource "helm_release" "istiod" {
  repository       = local.istio_charts_url
  chart            = "istiod"
  name             = "istiod"
  namespace        = kubernetes_namespace.istio_system.metadata.0.name
  version          = ">= 1.12.1"
  timeout          = 120
  cleanup_on_fail  = true
  force_update     = false
  
  set {
    name = "meshConfig.accessLogFile"
    value = "/dev/stdout"
  }
   
  depends_on       = [helm_release.istio-base]
}
    
resource "helm_release" "istio-ingress" {
  repository        = local.istio_charts_url
  chart             = "gateway"
  name              = "istio-ingress"
  namespace         = kubernetes_namespace.istio_system.metadata.0.name
  version           = ">= 1.12.1"
  timeout           = 500
  cleanup_on_fail   = true
  force_update      = false
  depends_on        = [helm_release.istiod]
}

PS: Please make sure you enable ports 15017 and 15021 in the master firewall rule for the istio ingress pod to properly start.

Answer 5

Just my 2 cents since i have tried all of the above solutions but nothing really suited me because i needed port 31400 to be exposed for grpc connection and in the way described above it is not feasible. I did the following

# var.istio_version = 1.17.2 in variables.tf
resource "null_resource" "dl_istio" {

  provisioner "local-exec" {
    command = "curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${var.istio_version} sh -"
  }
  depends_on = [ azurerm_kubernetes_cluster.default ] # cluster set up
}

resource "null_resource" "install_istio" {

  provisioner "local-exec" {
    command = "istio-${var.istio_version}/bin/istioctl install --set profile=demo -y"
  }
  depends_on = [ null_resource.dl_istio ]
}

data "kubernetes_service" "istio-ingressgateway" {
  metadata {
    name      = "istio-ingressgateway"
    namespace = "istio-system"
  }
  depends_on = [ null_resource.install_istio ]
}

Also i set up all the standard addons and exposed them using the services described in this link. The files expose_kiali.yaml, expose_grafana.yaml, expose_jaeger.yaml, expose_prometheus.yaml all have the copy pasted services as described in the link above.

The terraform configuration used was the following:

resource "null_resource" "install_istio_addons" {

  provisioner "local-exec" {
    command = "kubectl apply -f istio-${var.istio_version}/samples/addons"
  }
  depends_on = [ null_resource.install_istio ]
}

resource "local_file" "get_vars_kiali" {

  content = templatefile("${path.module}/addons/expose_kiali.yaml", {
    INGRESS_DOMAIN = "${data.kubernetes_service.istio-ingressgateway.status.0.load_balancer.0.ingress.0.ip}.nip.io"
  })
  filename = "${path.module}/addons/expose_kiali.yaml"
  depends_on = [ data.kubernetes_service.istio-ingressgateway ]
}

resource "null_resource" "expose_kiali" {
  provisioner "local-exec" {
    command = "kubectl apply -f ${local_file.get_vars_kiali.filename}"
  }
  depends_on = [ local_file.get_vars_kiali ]
}

resource "local_file" "get_vars_grafana" {

  content = templatefile("${path.module}/addons/expose_grafana.yaml", {
    INGRESS_DOMAIN = "${data.kubernetes_service.istio-ingressgateway.status.0.load_balancer.0.ingress.0.ip}.nip.io"
  })
  filename = "${path.module}/addons/expose_grafana.yaml"
  depends_on = [ data.kubernetes_service.istio-ingressgateway ]
}

resource "null_resource" "expose_grafana" {
  provisioner "local-exec" {
    command = "kubectl apply -f ${local_file.get_vars_grafana.filename}"
  }
  depends_on = [ local_file.get_vars_grafana ]
}


resource "local_file" "get_vars_jaeger" {

  content = templatefile("${path.module}/addons/expose_jaeger.yaml", {
    INGRESS_DOMAIN = "${data.kubernetes_service.istio-ingressgateway.status.0.load_balancer.0.ingress.0.ip}.nip.io"
  })
  filename = "${path.module}/addons/expose_jaeger.yaml"
  depends_on = [ data.kubernetes_service.istio-ingressgateway ]
}

resource "null_resource" "expose_jaeger" {
  provisioner "local-exec" {
    command = "kubectl apply -f ${local_file.get_vars_jaeger.filename}"
  }
  depends_on = [ local_file.get_vars_jaeger ]
}


resource "local_file" "get_vars_prometheus" {

  content = templatefile("${path.module}/addons/expose_prometheus.yaml", {
    INGRESS_DOMAIN = "${data.kubernetes_service.istio-ingressgateway.status.0.load_balancer.0.ingress.0.ip}.nip.io"
  })
  filename = "${path.module}/addons/expose_prometheus.yaml"
  depends_on = [ data.kubernetes_service.istio-ingressgateway ]
}

resource "null_resource" "expose_prometheus" {
  provisioner "local-exec" {
    command = "kubectl apply -f ${local_file.get_vars_prometheus.filename}"
  }
  depends_on = [local_file.get_vars_prometheus]
}

Hope it helps someone save some time.

Answer 6

Anyone has issues with installing the ingress gateway? The auto field does not seem to get populated with any real info. But also cannot even find a gateway image to run a docker image pull command on it. Anyone knows whatsapp?

The deployment yaml in the helm chart says this:

        - name: istio-proxy
          # "auto" will be populated at runtime by the mutating webhook. See https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#customizing-injection
          image: auto






  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m35s                default-scheduler  Successfully assigned istio-system/istio-ingress-54464bcb87-6kzbr to 10.137.111.166
 
 Normal   Pulling    67s (x4 over 2m33s)  kubelet            Pulling image "auto"
 
 Warning  Failed     66s (x4 over 2m30s)  kubelet            Failed to pull image "auto": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/auto:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:3606e91685fe202e729298edf590a9d58e202084d3f50b915c9a8a405e71ec4d: not found
  
Warning  Failed     66s (x4 over 2m30s)  kubelet            Error: ErrImagePull
 
 Warning  Failed     42s (x6 over 2m30s)  kubelet            Error: 
ImagePullBackOff

  Normal   BackOff    29s (x7 over 2m30s)  kubelet            Back-off pulling image "auto"