3

I set up a Load Balancer on the GCP using an Ingress in the Kubernetes Engine.

The Backend Service by default registered all instances as unhealthy. I updated the health check to the HTTPS protocol and the /healthz url, since that's the url for my health check.

I leave it alone for a couple of minutes and the backend service states all instances are healthy now, then I see in the logs that my pod is serving 200 responses at the /healthz.

I navigate to the page, and it starts showing a 502 error. I go back to the health check page in the console, and the health check reverted to HTTP at /.

What caused the health check to revert back to HTTP? Here's the logs showing that for 10 mins the health check continues to request the https I set, then the request changes back to http.

...
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.14', 48346): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.13', 63030): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.75ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.13) 0.80ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.74ms
...
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.13) 1.26ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.14) 0.63ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.15) 0.64ms
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.14', 49971): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.15', 62893): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 52191): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 60549): [SSL: HTTP_REQUEST] http request (_ssl.c:833)

EDIT: I'm adding the readiness probe for the pod in question, that the ingress is attached to.

readinessProbe:
  httpGet:
    port: 8902
    scheme: HTTPS
    path: /healthz
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1
lamorach
  • 31
  • 1
  • 3
  • 1
    Hi lamorach, I didn't understand the sentence "So, I continue trying to debug the instance then the health check reverts back to HTTP at the / url." Moreover I am not sure if you are interested to know why the instances were unhealthy or why the health check came back to HTTP. Can you clarify? – GalloCedrone Jun 15 '18 at 08:33
  • What caused the health check to revert back to HTTP? – lamorach Jun 15 '18 at 12:53
  • Usually it takes around 5 minutes to get all the load balancer set up, Probably your change never took efect if you made that change in couple of minutes. – suren Jun 16 '18 at 19:41
  • @suren I've updated my request log for your review. – lamorach Jun 21 '18 at 20:27
  • 1
    @lamorach, I have the same issue. Is this with k8s-generated checks that have a name like `k8s-be-30000--79a389181e59424e`? – alexferl Jun 21 '18 at 20:52
  • 1
    @limitedquantities Yes, and it's frustrating as can be because if you replace the generated checks with ones you make they flip back to the generated. – lamorach Jun 22 '18 at 12:52

3 Answers3

2

I checked and it shouds an intended behaviour.

Kubernetes controls the health check and configures it from data inside the kubernetes cluster. Kubernetes does not see any changes made to the health check via gcloud.

Therefore I guess it is normal that it checks it and any modification get reverted.

The solution to use readinessProbe instead.

Please take a look also to this:

Health checks

Currently, all service backends must satisfy either of the following requirements to pass the HTTP(S) health checks sent to it from the GCE loadbalancer:

  1. Respond with a 200 on '/'. The content does not matter.

  2. Expose an arbitrary URL as a readiness probe on the pods backing the Service.

The Ingress controller looks for a compatible readiness probe first, if it finds one, it adopts it as the GCE loadbalancer's HTTP(S) health check. If there's no readiness probe, or the readiness probe requires special HTTP headers, the Ingress controller points the GCE loadbalancer's HTTP health check at '/'.

GalloCedrone
  • 371
  • 1
  • 9
  • I've posted the readiness probe for the backend pod. It however is still attempting http port 80 / healthcheck. – lamorach Jun 27 '18 at 13:58
  • The health check will carry on checking /, the rediness probe will check your custom path – GalloCedrone Jul 06 '18 at 09:45
  • but if it continues to check just the slash, won't it still not define as ready because it will keep getting 404's? – lamorach Jul 10 '18 at 18:14
  • No, it will check the "/" merely if there is no rediness probe compatible "The Ingress controller looks for a compatible readiness probe first, if it finds one, it adopts it as the GCE loadbalancer's HTTP(S) health check. If there's no readiness probe, or the readiness probe requires special HTTP headers, the Ingress controller points the GCE loadbalancer's HTTP health check at '/'" – GalloCedrone Jul 16 '18 at 08:45
  • Until this is customizable in code, I found it's possible to modify the generated Health Check to use a Host header and custom path. Those changes don't get reverted (whereas assigning a different HC to the Backend does). (GKE 1.14 fwiw). https://github.com/kubernetes/ingress-gce/issues/937#issuecomment-555707740 – JCotton Nov 20 '19 at 21:29
1

I had the same issue.

The issue was that my readinessProbe was set to HTTPS only

readinessProbe:
  httpGet:
    path: "/health?ready=1"
    port: 8080
    scheme: HTTPS

However GCE Backends automatically assume that your services respond to HTTP and by default will scan for health checks there.

You can read more about GKE Ingresses and how Heath Checks respond HERE

To fix this I needed to update my service to respond with only HTTPS

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"my-port-name":"HTTPS"}'
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: my-port-name

This results in the backend service that the ingress creates to scan for readinessProbes over HTTPS.

NOTE: Be sure to tear down your Ingress, LoadBalancer, and HealthCheck before doing this then recreate it after as it will not respond to updates applied to your service.

wintheday
  • 11
  • 1
0

Looks like your backends are serving only on HTTPS protocol but you've deployed an Ingress resource for HTTP protocol. Manually changing the configuration after deployment is not the solution as the master would revert the changes to comply with the manifest. Per this document you can secure an Ingress by specifying a secret that contains a TLS private key and certificate. Currently the Ingress only supports a single TLS port, 443, and assumes TLS termination.

3ec
  • 1
  • can you specify /healthz for the health check somewhere, or have they not set that up either for the ingress? – lamorach Jun 26 '18 at 17:07
  • Also, I have an ingress with the tls defined, but the health check is still reverting to http. – lamorach Jun 26 '18 at 17:10