0

We are trying to find out why there is a high latency from Google CDN.

Our site is behind Google'a http_load_balancer with CDN turned on.

For example, by inspecting sampe GET request for a jpg file (43Kb), we can see from http_load_balancer logs that around 30% of such requests have httpRequest.latency > 1 second, and a lot are taking much longer like several or hundreds of seconds....

This is just by looking at 24h log sample (around 6K the same requests).

The httpRequest.cacheLookup and httpRequest.cacheHit for all of those requests are true. Also jsonpayload_type_loadbalancerlogentry.statusdetails is response_from_cache and jsonpayload_type_loadbalancerlogentry.cacheid values shows correct region.

When doing the same GET request manually in the browser we are getting expected results with TTFB around 15-20ms.

Any idea where to look for a clue?

bogumbiker
  • 437
  • 2
  • 5
  • 19
  • Check this [troubleshooting](https://cloud.google.com/cdn/docs/troubleshooting-steps) guide. – Mahboob Dec 20 '20 at 17:59
  • Thanks just had a look but I did not find anything around latency. – bogumbiker Dec 20 '20 at 22:31
  • In order to optimizing application latency with LB, you can follow this [doc](https://cloud.google.com/load-balancing/docs/tutorials/optimize-app-latency#optimizing) – Mahboob Dec 21 '20 at 00:08

2 Answers2

1

The httpRequest.latency field measures the entire download duration, and is directly impacted by slow clients - e.g. a mobile device on a congested network or throttled data plan.

You can check this by looking at the frontend_tcp_rtt metric (which is the RTT between the client and Cloud CDN) in Cloud Monitoring, as well as the average, median and 90th percentile total_latencies, where the slow clients will show up as outliers: https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#monitoring_metrics_fors

You may find that slow clients are from a specific group of client_country values.

elithrar
  • 23,364
  • 10
  • 85
  • 104
  • thanks for this info. I had a look here [httpRequest.latency](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#httprequest) and not sure `httpRequest.latency` measures the entire download duration. Is that field calculated based on final response from the client? Is there any metrics that shows how my site is performing at LoadBalancer (`http_load_balancer`) level in terms of response time? – bogumbiker Dec 22 '20 at 05:00
  • The metrics will give you performance from client <-> LB - e.g. round-trip-time (RTT) on the TCP connection, rather than the final response duration. This is also much easier to look at in aggregate vs. log data - https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing – elithrar Jan 19 '21 at 22:17
0

Latency can be introduced:

  • Between the original client and the load balancer. You can see the latency of that segment with the metric https/frontend_tcp_rtt.
  • Between the load balancer and the backend instance. Which can be reviewed with the metric https/backend_latencies (this metric also includes the app processing time in your backend).
  • By the software running on the instance itself. To investigate this I would check the access/error logs on the backend instance software and resource utilization of the VM instance.

Further information about metrics description on the GCP load balancer metrics doc.

httpRequest.latency log field description: "The request processing latency on the server, from the time the request was received until the response was sent."

Oscar DP
  • 26
  • 1