Configuring the Health Check of a Kubernetes Ingress with Terraform

Question

We are using an ingress (kubernetes_ingress.db_admin_ingress) to expose the service (kubernetes_service.db_admin) of a deployment (kubernetes_deployment.db_admin) in Google Kubernetes Engine (GKE) with Terraform.

When Terraform creates the ingress, a Level 7 Load Balancer is automatically created with a default health check:

• port: 80
• path: /
• protocol: HTTP(S)

Our deployment (kubernetes_deployment.db_admin) does not respond to the path / with a 200, so the health check fails.

How can we change the path in the health check configuration?

resource "google_compute_managed_ssl_certificate" "db_admin_ssl_certificate" {
  provider = google-beta

  name = "db-admin-ssl-certificate"

  managed {
    domains = ["db.${var.domain}."]
  }
}

resource "kubernetes_deployment" "db_admin" {
  metadata {
    name = "db-admin"
    labels = {
      App = "db-admin"
    }
  }

  spec {
    replicas = 1
    selector {
      match_labels = {
        App = "db-admin"
      }
    }
    template {
      metadata {
        labels = {
          App = "db-admin"
        }
      }
      spec {
        container {
          image = "dpage/pgadmin4:2022-01-10-1"
          name  = "db-admin"
          env {
            name = "PGADMIN_DEFAULT_EMAIL"
            value = "test@test.com"
          }
          env {
            name = "PGADMIN_DEFAULT_PASSWORD"
            value = "test"
          }      

          port {
            container_port = 80
          }

          resources {}
        }
      }
    }
  }
}

resource "kubernetes_service" "db_admin" {
  metadata {
    name = "db-admin"
  }
  spec {
    selector = {
      App = kubernetes_deployment.db_admin.spec.0.template.0.metadata[0].labels.App
    }
    port {
      protocol    = "TCP"
      port        = 80
      target_port = 80
    }

    type = "NodePort"
  }
}

resource "kubernetes_ingress" "db_admin_ingress" {
  wait_for_load_balancer = true
  
  metadata {
    name = "db-admin-ingress"
    annotations = {
      "ingress.gcp.kubernetes.io/pre-shared-cert"   = google_compute_managed_ssl_certificate.db_admin_ssl_certificate.name
    }
  }

  spec {

    rule {
      http {
        path {
          backend {
            service_name = "db-admin"
            service_port = 80
          }

          path = "/*"
        }


      }
    }

  }
}

Answer 1

According to Google Kubernetes Engine (GKE) official documentation here, you are able to customize ingress/Level 7 Load Balancer health checks through either:

• the readinessProbe for the container within the pod your ingress is serving traffic to

  Warning: this method comes with warnings here

• a backendconfig resource

I would highly recommend creating a backendconfig resource.

Unfortunately, the kubernetes Terraform provider does not seem to support the backendconfig resource based on this GitHub issue. This means that you can either:

• use the kubernetes-alpha provider (found here) to transcribe a YAML backendconfig manifest to HCL with the manifest argument for the only kubernetes-alpha resource: kubernetes-manifest (more on that here)
• use an unofficial provider (such as banzaicloud/k8s found here)
• check the backendconfig manifest (as either JSON or YAML) into SCM

A sample backendconfig YAML manifest:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: db-admin
  namespace: default
spec:
  healthCheck:
    checkIntervalSec: 30
    timeoutSec: 5
    healthyThreshold: 1
    unhealthyThreshold: 2
    type: HTTP
    requestPath: /v1/some/path
    port: 80

Note: a service is needed to associate a backendconfig with an ingress/Level 7 Load Balancer:

apiVersion: v1
kind: Service
metadata:
  name: db-admin-ingress-backend-config
  labels:
    app: db-admin
  annotations:
    cloud.google.com/backend-config: '{"ports": {"80":"db-admin"}}'
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: NodePort
  selector:
    app: db-admin
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80

You can learn more about the backendconfig resource and the service it requires here.