6

The problem I am trying to solve:

I want to achieve the following setup:

  1. Terraform provisioned infrastructure consisting of Kubernetes cluster, DNS records pointing to that cluster,
  2. Kubernetes services, deployments configured through Kubernetes .yaml files,
  3. Kubernetes ingress service acting as a gateway to the services on the cluster.

How I am trying to do this:

  1. Add kubernetes cluster and kubernetes ingress through Terraform. Bind kubernetes ingress IP to DNS records.
  2. Deploy all the services with kubectl.

The relevant part of terraform code for this looks something like this:

resource "kubernetes_ingress" "main-ingress" {
  metadata {
    name = "main-ingress"
  }

  spec {
    rule {
      host = "service.example.com"
      http {
        path {
          backend {
            service_name = "service-name"
            service_port = 8080
          }

          path = "/"
        }
      }
    }
  }

  wait_for_load_balancer = true
}

resource "digitalocean_record" "subdomain_link" {
  domain = "example.com"
  type   = "A"
  name   = "service"
  value  = kubernetes_ingress.main-ingress.load_balancer_ingress[0].ip
}

The problem is that kubernetes_ingress does not seem to be able to be deployed without managing the pods, services and deployments through Terraform. I want to manage them separately through kubectl kustomize files.

The reason I am trying to create kubernetes ingress through Terraform is that I need the assigned IP for the DNS records in terraform.

The question I have is as follows:

Can I deploy kubernetes ingress through terraform without managing the kubernetes service through terraform?

If I can't, is there any way to know what IP will be assigned to the ingress during terraform apply so that I can bind the DNS to it?

P.S. Sorry for the convoluted question, not sure how I can structure it better.

Adam
  • 1,470
  • 1
  • 17
  • 13
  • 1
    I don't think there is such a limitation. I tried and it worked. `main-ingress service.example.com 192.168.64.17 80 33s` What's the error you're getting? – hariK May 21 '20 at 19:46
  • First kubernetes apply ends with what seems like timeout: `kubernetes_ingress.main-ingress: Still creating... [20m0s elapsed] Error: Load Balancer is not ready yet 42: resource "kubernetes_ingress" "main-ingress" { `When I am doing second apply I get the following error: `Error: Invalid index: kubernetes_ingress.main-ingress.load_balancer_ingress is empty list of object; The given key does not identify an element in this collection value.` I concluded that this is not possible. Maybe there is something wrong with configuration? Or something connected with DO hosting itself. – Adam May 21 '20 at 20:12
  • 1
    It failed because of the unavailability of the LB ATM. But still, you can create an ingress resource alone. – hariK May 21 '20 at 20:16
  • Thank you for the response. I am still struggling with understanding. What do you mean by the unavailability of LB at the moment? How can ingress not be available if I am setting it up on my own cluster that is ready? I thought the first one failed because it was looking for the "service-name" service and couldn't find it. – Adam May 21 '20 at 20:20
  • I also found out that this problem may not occur because the LB is not available, but because the service to which the ingress is routing to does not exist at the moment of creating, therefore it errors and remains in a creating loop. Make sure both the service and the pod exist and use the `wait_for_load_balancer = true`. At least this worked for me. – toaster_fan Nov 17 '20 at 09:32
  • Were you able to solve this problem ? – Mark Mar 15 '21 at 18:50
  • @Mark No I wasn't able to solve this. – Adam Mar 16 '21 at 05:13
  • @Adam Do you have an ingress controller in your cluster? – Marko E May 26 '22 at 08:34
  • Do you _have_ to manage the DNS record with Terraform or could you deploy the external dns component to your cluster? https://github.com/kubernetes-sigs/external-dns If there are security concerns allowing the cluster to modify the DNS zone that you are using, create a new zone just for your cluster to play in, and create CNAMEs in Terraform from the real domain names. – Sharebear Jun 10 '23 at 18:44

0 Answers0