client disconnected before any response GCLB

Question

We deployed our site in front GCLB.

LB -> Cloud run -> APP ENGINE API

Cloud run is hosting a react site and App Engine golang API.

After 12 hours we started to saw decline in the amount of clicks via google analytics but traffic was pretty much the same.

Our assumption is that "lost" traffic somehow, I can see in logs 2 main issue.

404 with address of old site components.
client disconnected before any response error.

I can understand the 404 error its cache request that looking for old site components.

But i don`t understand client disconnected error and if its related to our "lost" traffic.

Any suggestion how to analyze our "lost" traffic?

UPDATE:

I found some correlation to the client client disconnected error.

The requestUrl contains images resources for exemple images/zoom.png?v1.0
Back end service name is empty backend_service_name: "" not sure how it can be empty, I mapped all the resources and host

LOG

{
  "insertId": "cs2fmdg2eo8nba",
  "jsonPayload": {
    "cacheId": "FRA-1209ea83",
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "client_disconnected_before_any_response"
  },
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "https://travelpricedrops.com/images/aero.png?v1.0",
    "requestSize": "78",
    "userAgent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_8 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1",
    "remoteIp": "109.104.52.1",
    "referer": "https://travelpricedrops.com/passthru?tab=front&vert=flights&origin-iata=LEJ&destination-iata=JFK&departure-time=2021-12-26T11%3A00%3A00Z&cabin-class=economy&num-adults=1&num-youth=0&rental-duration=6&dta=48&return-time=2022-01-01T11%3A00%3A00Z&f=cf&fuid=1102&b=k&buid=1043",
    "cacheLookup": true,
    "latency": "0.071958s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "zone": "global",
      "backend_service_name": "",
      "forwarding_rule_name": "tpd-int-https-ipv4",
      "target_proxy_name": "int-tpd-target-proxy-2",
      "url_map_name": "int-tpd",
      "project_id": "tpdrops"
    }
  },
  "timestamp": "2021-11-09T06:13:55.121455Z",
  "severity": "INFO",
  "logName": "projects/tpdrops/logs/requests",
  "trace": "projects/tpdrops/traces/13821ba38ae9e3191381f3f64b0a7b1a",
  "receiveTimestamp": "2021-11-09T06:13:55.343086132Z",
  "spanId": "a5ae86336a24bc32"
}

Config

**gcloud compute forwarding-rules describe tpd-int-https-ipv4**

IPAddress: 34.149.93.11
IPProtocol: TCP
creationTimestamp: '2021-08-30T11:49:06.047-07:00'
description: ''
fingerprint: CIAg3TcEb9Y=
id: '1815919129513727693'
kind: compute#forwardingRule
labelFingerprint: 42WmSpB8rSM=
loadBalancingScheme: EXTERNAL
name: tpd-int-https-ipv4
networkTier: PREMIUM
portRange: 443-443
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/forwardingRules/tpd-int-https-ipv4
target: https://www.googleapis.com/compute/v1/projects/tpdrops/global/targetHttpsProxies/int-tpd-target-proxy-2

**gcloud compute backend-services describe tpd-prod-back**
affinityCookieTtlSec: 0
backends:
- balancingMode: UTILIZATION
  capacityScaler: 0.0
  group: https://www.googleapis.com/compute/v1/projects/tpdrops/regions/us-central1/networkEndpointGroups/tpd-front
cdnPolicy:
  cacheKeyPolicy:
    includeHost: true
    includeProtocol: true
    includeQueryString: true
  cacheMode: CACHE_ALL_STATIC
  clientTtl: 3600
  defaultTtl: 3600
  maxTtl: 86400
  negativeCaching: false
  requestCoalescing: true
  serveWhileStale: 86400
  signedUrlCacheMaxAgeSec: '0'
connectionDraining:
  drainingTimeoutSec: 0
creationTimestamp: '2021-10-25T04:09:29.908-07:00'
description: ''
enableCDN: true
fingerprint: 5FNZk6GXJTw=
iap:
  enabled: false
id: '6357784085114072710'
kind: compute#backendService
loadBalancingScheme: EXTERNAL
logConfig:
  enable: true
  sampleRate: 1.0
name: tpd-prod-back
port: 80
portName: http
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
sessionAffinity: NONE
timeoutSec: 30

**gcloud compute url-maps describe int-tpd**
creationTimestamp: '2021-08-29T06:08:35.918-07:00'
defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
fingerprint: trtG9xBMlvE=
hostRules:
- hosts:
  - acpt.travelpricedrops.com
  pathMatcher: path-matcher-2
- hosts:
  - int.travelpricedrops.com
  pathMatcher: path-matcher-1
- hosts:
  - api.acpt.travelpricedrops.com
  pathMatcher: path-matcher-3
- hosts:
  - api.int.travelpricedrops.com
  pathMatcher: path-matcher-4
- hosts:
  - api.travelpricedrops.com
  pathMatcher: path-matcher-5
- hosts:
  - travelpricedrops.com
  pathMatcher: path-matcher-6
id: '6018005644614187068'
kind: compute#urlMap
name: int-tpd
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-acpt-back
  name: path-matcher-2
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-int-http
  name: path-matcher-1
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api-acpt
  name: path-matcher-3
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api-int
  name: path-matcher-4
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api
  name: path-matcher-5
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
  name: path-matcher-6
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/urlMaps/int-tpd


 **gcloud compute target-http-proxies describe int-tpd-target-proxy-2**
ERROR: (gcloud.compute.target-http-proxies.describe) Could not fetch resource:
 - The resource 'projects/tpdrops/global/targetHttpProxies/int-tpd-target-proxy-2' was not found

yep lost traffic isn't to best phrase... for this use case is traffic that arrive from goggle ads to black hole... — MIkCode, Nov 10 '21 at 10:14
Please update your question with your LB's detailed configuration. Please also include the logs you mentioned (if possible). — Wojtek_B, Nov 10 '21 at 13:16
@Wojtek_B i cant find the command to export configuration but added logs — MIkCode, Nov 10 '21 at 13:40
You need to use several commands: [`gcloud compute backend-services describe service_name_here`](https://cloud.google.com/sdk/gcloud/reference/compute/backend-services/describe), [`gcloud compute forwarding-rules describe tpd-int-https-ipv4`](https://cloud.google.com/sdk/gcloud/reference/compute/forwarding-rules/describe), [`gcloud compute target-http-proxies describe nt-tpd-target-proxy-2`](https://cloud.google.com/sdk/gcloud/reference/compute/target-http-proxies/describe) and [`gcloud compute url-maps describe int-tpd`](https://cloud.google.com/sdk/gcloud/reference/compute/url-maps/describe) — Wojtek_B, Nov 10 '21 at 14:37
@Wojtek_B also it look like i can fetch "int-tpd-target-proxy-2" — MIkCode, Nov 10 '21 at 15:24
Try `gcloud compute target-https-proxies describe nt-tpd-target-proxy-2` instead. — Wojtek_B, Nov 12 '21 at 10:00

score 1 · Accepted Answer · answered Nov 12 '21 at 12:13

Your load balancer's configuration looks ok; you have a https-ssl-secured frontend on port 443 pointing to a http backend on port 80 which means that SSL is resolved at the load balancer and sent in plain http to your backend.

Error you're getting means (as per documentation) that the client disconnected before load balancer could reply:

client_disconnected_before_any_response - The connection to the client was broken before the load balancer sent any response.

Now to answer your questions.

Since the images are served directly by your app (I didn't see any host-path rules saying otherwise) make sure that application can serve images in time. Set your application response timeout to 10 seconds or more and this should solve the issue. Have a look at this discussion which may be quite usefull for you. 1.1 - there's also a configurable request timeout for Cloud Run services - you can check it by running gcloud run services describe SERVICE_NAME
The backend_service_name: "" string you mentioned may be empty - nothing to worry about - this is an expected behavior.

Additionally have a look at the Backend service timeout Timeouts and retries in external load balancing which may also put some light onto your case.

Lastly - have a look at How to debug failed requests with client_disconnected_before_any_response.

@WytrzymałyWiktor I still not sure because time out on cloud run is set to ***Timeout: 300s***. So i dont know how to solve it. — MIkCode, Nov 15 '21 at 09:34
What are the sizes of the images that are served ? Maybe for some reason it takes you app too long to respond ? — Wojtek_B, Nov 15 '21 at 09:56
@Wojtek_B most of them under 100kb and the largest one is 490kb — MIkCode, Nov 16 '21 at 09:21

client disconnected before any response GCLB

1 Answers1