Problem creating regional TCP load balancer

Question

I'm trying to create a load-balancer that balances traffic between 3 different AZ's in a given region. If I create a "global" load-balancer with an external IP, everything works fine, but if I am only trying to create a load-balancer that works with a particular subnet -- the health checks consistently fail because they are trying to go to port 80 instead of the port I've specified.

Note the following output of gcloud compute backend-services get-health xx-redacted-central-lb --region=us-central1:

---
backend: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-a/instanceGroups/xx-redacted-central-a
status:
  healthStatus:
  - healthState: UNHEALTHY
    instance: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-a/instances/yb-1-xx-redacted-lb-test-n2
    ipAddress: 10.152.0.90
    port: 80
  kind: compute#backendServiceGroupHealth
---
backend: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-b/instanceGroups/ac-kroger-central-b
status:
  healthStatus:
  - healthState: UNHEALTHY
    instance: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-b/instances/yb-1-xx-redacted-lb-test-n1
    ipAddress: 10.152.0.92
    port: 80
  kind: compute#backendServiceGroupHealth
---
backend: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-c/instanceGroups/xx-redacted-central-c
status:
  healthStatus:
  - healthState: UNHEALTHY
    instance: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-c/instances/yb-1-xx-redacted-lb-test-n3
    ipAddress: 10.152.0.4
    port: 80
  kind: compute#backendServiceGroupHealth

The health-check for this load-balancer was created with the following command: gcloud compute health-checks create tcp xx-redacted-central-hc4 --port=5433

The backend was created like this: gcloud compute backend-services create xx-redacted-central-lb --protocol=TCP --health-checks=xx-redacted-central-hc4 --region=us-central1 --load-balancing-scheme=INTERNAL

Full description of the backend: gcloud compute backend-services describe xx-redacted-central-lb --region=us-central1

backends:
- balancingMode: CONNECTION
  group: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-a/instanceGroups/xx-redacted-central-a
- balancingMode: CONNECTION
  group: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-b/instanceGroups/xx-redacted-central-b
- balancingMode: CONNECTION
  group: https://www.googleapis.com/compute/v1/projects/yugabyte/zones/us-central1-c/instanceGroups/xx-redacted-central-c
connectionDraining:
  drainingTimeoutSec: 0
creationTimestamp: '2020-04-01T19:16:44.405-07:00'
description: ''
fingerprint: aOB7iT47XCk=
healthChecks:
- https://www.googleapis.com/compute/v1/projects/yugabyte/global/healthChecks/xx-redacted-central-hc4
id: '1151478560954316259'
kind: compute#backendService
loadBalancingScheme: INTERNAL
name: xx-redacted-central-lb
protocol: TCP
region: https://www.googleapis.com/compute/v1/projects/yugabyte/regions/us-central1
selfLink: https://www.googleapis.com/compute/v1/projects/yugabyte/regions/us-central1/backendServices/xx-redacted-central-lb
sessionAffinity: NONE
timeoutSec: 30

If I try to edit the backend and add a port or portname annotation, it fails to save because thinks it is an invalid operation with INTERNAL load-balancers.

Any ideas? --Alan

Internal TC/UP load balancers always forward the same port. Therefore the port mapping attributes of the backend can be ignored. I suppose this only shows 80, because it's the default. Concerning health, what is your backend like? Does it listen to all network interfaces? Do you have the routing configured correctly? — Laurens Knoll, Apr 01 '22 at 11:52

score 1 · Answer 1 · answered Apr 04 '20 at 00:46

1

As per GCP documentation[1], For health checks to work, you must create an ingress to allow firewall rules for the ip address traffic from Google Cloud probers can connect to your backends.

You can review this documentation[2] to understand the Success criteria for SSL and TCP health check.

[1]Probe IP ranges and firewall rules https://cloud.google.com/load-balancing/docs/health-check-concepts#ip-ranges

[2]Success Criteria https://cloud.google.com/load-balancing/docs/health-check-concepts#criteria-protocol-ssl-tcp

answered Apr 04 '20 at 00:46

Dattu Pragnu Nellutla

21
4

That really doesn't address the issue at hand. I have set up the proper firewall rules as indicated by the documentation. My issue is that I clearly defined the **TCP** health check to operate on port **5433**, but it is trying to conduct the health check on port 80. – Alan Apr 05 '20 at 14:28

GagandeepT · Answer 2 · 2020-04-06T21:14:46.487

0

Backend services must have an associated Named Port if their backends are instance groups. Named ports are used by load balancing services to direct traffic to specific ports on individual instances. You can assign port name mapping to Instance group, to inform the load balancer to use that port to reach to backend running the service.

edited Apr 06 '20 at 21:14

answered Apr 06 '20 at 21:07

GagandeepT

260
1
10

Indeed there are named ports for each unmanaged instance group. Still don't understand why the healthcheck is checking port 80. – Alan Apr 07 '20 at 23:26

score 0 · Answer 3 · answered Apr 09 '20 at 02:01

Thanks for providing the information. I can successfully reproduce this issue at my end and find it strange that backend health checks are still being pointed to port 80 whereas LB HC configured for a port other than 80. The product engineering team has been made aware of this issue however, I don't have any ETA on the fix and implementation. You may follow thread[1] for further updates.

[1]https://issuetracker.google.com/153600927

Problem creating regional TCP load balancer

3 Answers3