0

I have on the backend a Kubernetes node running on port 32656 (Kubernetes Service of type NodePort). If I create a firewall rule for the <node_ip>:32656 to allow traffic, I can open the backend in the browser on this address: http://<node_ip>:32656.

What I try to achieve now is creating an HTTP Load Balancer and link it to the above backend. I use the following script to create the infrastructure required:

#!/bin/bash

GROUP_NAME="gke-service-cluster-61155cae-group"
HEALTH_CHECK_NAME="test-health-check"
BACKEND_SERVICE_NAME="test-backend-service"
URL_MAP_NAME="test-url-map"
TARGET_PROXY_NAME="test-target-proxy"
GLOBAL_FORWARDING_RULE_NAME="test-global-rule"
NODE_PORT="32656"
PORT_NAME="http"

# instance group named ports
gcloud compute instance-groups set-named-ports "$GROUP_NAME" --named-ports "$PORT_NAME:$NODE_PORT"

# health check
gcloud compute http-health-checks create --format none "$HEALTH_CHECK_NAME" --check-interval "5m" --healthy-threshold "1" --timeout "5m" --unhealthy-threshold "10"

# backend service
gcloud compute backend-services create "$BACKEND_SERVICE_NAME" --http-health-check "$HEALTH_CHECK_NAME" --port-name "$PORT_NAME" --timeout "30"
gcloud compute backend-services add-backend "$BACKEND_SERVICE_NAME" --instance-group "$GROUP_NAME" --balancing-mode "UTILIZATION" --capacity-scaler "1" --max-utilization "1"

# URL map
gcloud compute url-maps create "$URL_MAP_NAME" --default-service "$BACKEND_SERVICE_NAME"            

# target proxy
gcloud compute target-http-proxies create "$TARGET_PROXY_NAME" --url-map "$URL_MAP_NAME"

# global forwarding rule
gcloud compute forwarding-rules create "$GLOBAL_FORWARDING_RULE_NAME" --global --ip-protocol "TCP" --ports "80" --target-http-proxy "$TARGET_PROXY_NAME"

But I get the following response from the Load Balancer accessed through the public IP in the Frontend configuration:

Error: Server Error

The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

The health check is left with default values: (/ and 80) and the backend service responds quickly with a status 200.

I have also created the firewall rule to accept any source and all ports (tcp) and no target specified (i.e. all targets).

Considering that regardless of the port I choose (in the instance group), and that I get the same result (Server Error), the problem should be somewhere in the configuration of the HTTP Load Balancer. (something with the health checks maybe?)

What am I missing from completing the linking between the frontend and the backend?

Gabriel Petrovay
  • 20,476
  • 22
  • 97
  • 168
  • What is the actual HTTP response code? 5xx? – jarmod Jun 27 '16 at 14:54
  • Diagnose health checks? https://cloud.google.com/compute/docs/load-balancing/http/backend-service#health_checking – jarmod Jun 27 '16 at 17:48
  • I have increased all the timeouts and trial count before failure. And decreased the ones for success. Also I have checked that the health check URL returns a 200 status (written above as `/a_status_200_returning_URL`). See my updated code for the creation of the health check. The result is still the same 502. I have replicated again on a bigger cluster to make sure it is not the scarce resource that might cause a sluggish response. – Gabriel Petrovay Jun 28 '16 at 15:15
  • You didn't specify port on http-health-check. Does `/a_status_200_returning_URL` work on port 80 (a default one)? – Piotr Butryn Jun 29 '16 at 14:53
  • The backend works on port `80`, is is exposed by the node on port `32656`. So I left the default port (`80`) for the health check since, as far as I know the health check should be associated with the frontend (port `80`). Nevertheless, I made a try with the backend (`NodeType`) port (`32656`). No success. The same `502` error: `Server Error` – Gabriel Petrovay Jun 30 '16 at 09:17

3 Answers3

1

I assume you actually have instances in the instance group, and the firewall rule is not specific to a source range. Can you check your logs for a google health check? (UA will have google in it).

What version of kubernetes are you running? Fyi there's a resource in 1.2 that hooks this up for you automatically: http://kubernetes.io/docs/user-guide/ingress/, just make sure you do these: https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md.

More specifically: in 1.2 you need to create a firewall rule, service of type=nodeport (both of which you already seem to have), and a health check on that service at "/" (which you don't have, this requirement is alleviated in 1.3 but 1.3 is not out yet).

Also note that you can't put the same instance into 2 loadbalanced IGs, so to use the Ingress mentioned above you will have to cleanup your existing loadbalancer (or at least, remove the instances from the IG, and free up enough quota so the Ingress controller can do its thing).

Prashanth B
  • 4,833
  • 1
  • 19
  • 13
  • I am using K8s `1.2.4`. I have a firewall rule that allows port `80` from any source (range) to any target (tags). Previously I made also firewall rules with tags like instance id where the node is running, the node id (which happens to be now the same as the instance id), the instance group id. Always the same `502` error. – Gabriel Petrovay Jun 30 '16 at 09:56
  • Regarding the health check, I used another path (`/a_status_200_returning_URL`) but I also changed this to `/` in order to make sure this is not a limitation of the kubernetes liveness/readiness probes (not GCE L7 where you can choose a custom path using `--request-path`). – Gabriel Petrovay Jun 30 '16 at 10:37
  • Regarding the instances, I only use one single instance and one group. I am not using the same instances in more groups. – Gabriel Petrovay Jun 30 '16 at 10:39
0

There can be a few things wrong that are mentioned:

  • firewall rules need to be set to all hosts, are they need to have the same network label as the machines in the instance group have

  • by default, the node should return 200 at / - readiness and liveness probes to configure otherwise did not work for me

It seems you try to do things that are all automated, so I can really recommend: https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress

This shows the steps that do the firewall and portforwarding for you, which also may show you what you are missing.

I noticed myself when using an app on 8080, exposed on 80 (like one of the deployments in the example) that the load balancer staid unhealthy untill I had / returning 200 (and /healthz I added to). So basically that container now exposes a webserver on port 8080, returning that and the other config wires that up to port 80.

When it comes to firewall rules, make sure they are set to all machines or make the network label match, or they won't work. The 502 error is usually from the loadbalancer that will not pass your request if the health check does not pass.

Vincent Gerris
  • 7,228
  • 1
  • 24
  • 22
-1

Could you make your service type LoadBalancer (http://kubernetes.io/docs/user-guide/services/#type-loadbalancer) which would setup this all up automatically? This assumes you have the flag set for google cloud.

After you deploy, then describe the service name and should give you the endpoint which is assigned.

Steve Sloka
  • 3,444
  • 2
  • 22
  • 29
  • I want to avoid using a `LoadBalancer` (see http://stackoverflow.com/q/37792641/454103). 1. It makes no sense in our test use cases (we only have one node). 2. It costs significantly more when many services run in the same time (compared to a HTTP Load Balancer where you only have 1 Load Balancer for more applications that run behind it) – Gabriel Petrovay Jun 27 '16 at 16:45
  • 2
    Oh ok, that wasn't clear to me that you were making your own LB (I think I just misread). Sounds like you want to use ingress routing (http://kubernetes.io/docs/user-guide/ingress/) or roll your own which is ok since you're only on a single node. – Steve Sloka Jun 27 '16 at 17:10