2

After I create a very basic Ingress (yaml below) with no special definition of health checks (nor any in other Kubernetes objects), the Ingress creates a GCP Load Balancer.

Why does this LB have two health checks ("backend services") defined against different nodePorts, one against root path / and one against /healthz? I would expect to see only one.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress1
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: myservice
          servicePort: 80
Joshua Fox
  • 18,704
  • 23
  • 87
  • 147
  • When you say **health check ("backend services")**, you mean it's the same thing? It's weird the LB creates a health check on /healthz, out of nowhere. Can you share more details? – suren Jan 01 '20 at 16:31
  • Yes, there are two health checks on the k8s ingress which map directly to two backend services in the LB. And yes, it's weird. What kind of details might help? – Joshua Fox Jan 01 '20 at 16:40
  • Did you define a `livenessProbe` in the definition of the pod(s) exposed by `myservice`? – LundinCast Jan 01 '20 at 16:41
  • a `livenessProbe` is done by the `kubelet` it wouldn't create an external health check. – suren Jan 01 '20 at 17:00

1 Answers1

4

Reason for / health check

One of the limitation of GKE ingress controller is below:

For the GKE ingress controller to use your readinessProbes as health checks, the Pods for an Ingress must exist at the time of Ingress creation. If your replicas are scaled to 0 or Pods don't exist when the ingress is created, the default health check using / applies.

Because of the above it created two health checks.

There are lot of caveats for the health-check from readiness probe to work:

  • The pod's containerPort field must be defined
  • The service's targetPort field must point to the pod port's containerPort value or name. Note that the targetPort defaults to the port value if not defined
  • The pods must exist at the time of ingress creation
  • The readiness probe must be exposed on the port matching the servicePort specified in the Ingress
  • The readiness probe cannot have special requirements like headers The probe timeouts are translated to GCE health check timeouts

Based on above it makes sense to have a default fallback health check using /

Reason for /healthz health check

As per this FAQ all GCE URL maps require at least one default backend, which handles all requests that don't match a host/path. In Ingress, the default backend is optional, since the resource is cross-platform and not all platforms require a default backend. If you don't specify one in your yaml, the GCE ingress controller will inject the default-http-backend Service that runs in the kube-system namespace as the default backend for the GCE HTTP lb allocated for that Ingress resource.

Some caveats concerning the default backend:

  • It is the only Backend Service that doesn't directly map to a user specified NodePort Service
  • It's created when the first Ingress is created, and deleted when the last Ingress is deleted, since we don't want to waste quota if the user is not going to need L7 loadbalancing through Ingress
  • It has a HTTP health check pointing at /healthz, not the default /, because / serves a 404 by design

So GKE ingress deploys a default backend service using below yaml and setup a /healthz healthcheck for that service.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: l7-default-backend
  namespace: kube-system
  labels:
    k8s-app: glbc
    kubernetes.io/name: "GLBC"
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: glbc
  template:
    metadata:
      labels:
        k8s-app: glbc
        name: glbc
    spec:
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: k8s.gcr.io/defaultbackend-amd64:1.5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  # This must match the --default-backend-service argument of the l7 lb
  # controller and is required because GCE mandates a default backend.
  name: default-http-backend
  namespace: kube-system
  labels:
    k8s-app: glbc
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "GLBCDefaultBackend"
spec:
  # The default backend must be of type NodePort.
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    k8s-app: glbc
Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107
  • Thank you. Could you please send a link about "If your replicas are scaled to 0 or Pods don't exist when the ingress is created...". I experimented by building this simple ingress example in the right order as well as the wrong order, and either way, the health check is against / – Joshua Fox Jan 02 '20 at 07:50
  • 1
    Look at the Limitations section of https://cloud.google.com/kubernetes-engine/docs/concepts/ingress – Arghya Sadhu Jan 02 '20 at 07:54
  • Make sure that all the caveats I mentioned are covered for the /healthz to work – Arghya Sadhu Jan 02 '20 at 07:57
  • Thank you. So that is where the `/` check comes from. But where does the `healthz` come from? It does not appear in any of our k8s yamls (nor does any healthcheck or readinessProbe). – Joshua Fox Jan 02 '20 at 08:03
  • Well my understanding is that GKE ingress controller automatically creates health check at healthz in google cloud load balancer and expects you to configure a readiness in your pod at /healthz – Arghya Sadhu Jan 02 '20 at 10:21
  • These are relevant https://stackoverflow.com/questions/51869789/ https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-loadbalancing/glbc/default-svc-controller.yaml – Joshua Fox Jan 02 '20 at 10:36
  • Some work in progress to solve this holistically https://github.com/kubernetes/ingress-gce/issues/42 – Arghya Sadhu Jan 02 '20 at 12:09
  • Thank you, that is relevant. I'd still like to see documentation that says when a check at healthz is automatically added. – Joshua Fox Jan 02 '20 at 12:27
  • Check if you have a service "default-http-backend" in Kube-system namespace? – Arghya Sadhu Jan 02 '20 at 13:16
  • Yes, there is. Does this explain why we see a healthz heathcheck? – Joshua Fox Jan 02 '20 at 13:36
  • Thank you. How can "/ serves a 404 by design"? In that case, the default health check will always fail?! – Joshua Fox Jan 02 '20 at 18:46