1

I'm on 'basic support' plan with AWS, and cannot therefore ask the in-house technical experts there. I was trying to set up cloudfront. My backend is at AWS Hong Kong. However, I noticed that traffic that is 'front-ended' by Cloudfront HK servers do not go directly to my back-end EC2 server which is also in AWS Hong Kong. Instead, the request comes to my HK EC2 from AWS Singapore. (In other words, the HK CloudFront proxy fetches content from my HK EC2 origin server via Singapore.) Is this common? It's adding unnecessary latency to the request. A direct request can be completed in 100 ms, but this takes 250 ms. I checked if this was only with HK, and turns out it is not. I sent out a request from Tokyo, which was captured by CloudFront Tokyo. From there, instead of sending it directly to HK, it sent it to Singapore and from there, it was redirected back to HK. Any idea why this is happening? I cannot think of any logical reason why AWS would route traffic meant for another server within the same location via a third country, instead of routing it directly within the location.

For the record, the origin server is given in the classic aws DNS notation (ip-address-location-awz.amazon.com) within Cloudfront. There can't be a situation that CF think it's a non-AWS IP address.

Prashant D
  • 13
  • 2
  • I don't have it. As soon as I saw it was doing this, I switched it to Tokyo and then shifted the origin server to Singapore AWS. But I do have the domain for the HK server "ec2-18-163-63-38.ap-east-1.compute.amazonaws.com" . One funny thing I noticed was that from the US West Coast, the full curl request via CloudFront to the HK server was taking only 220 ms, while it was taking 400 from Tokyo, and about 500 from US East Coast. It's totally unpredictable. – Prashant D Oct 09 '19 at 13:33

1 Answers1

2

The reason for what you are seeing is documented, but there is not a lot of detail or discussion of the implications provided in the documentation.

The CloudFront network has two tiers.

There's the outer "global" tier and the inner "regional" tier. Most AWS regions have a regional CloudFront edge in them, but not all -- and Hong Kong is currently an exception, an AWS region where there is not a regional edge.

Requests typically arrive at a "global edge" location near the viewer (e.g. Atlanta, GA, US) where the edge cache is checked and the content is served, or the request is forwarded to the origin.

But it's not forwarded directly to the origin.

Instead, the request is forwarded to the nearest "regional cache" (e.g. from Atlanta, that's probably Ohio, US -- the us-east-2 region) where that cache is checked for a response. If found, it's returned to the global edge that handled the original request, where it's stored in the cache (if cacheable) and returned to the viewer.

If the regional cache doesn't have a copy, either, then the request is forwarded from there to the origin server, and the response is cached at the regional cache, and returned to the original global edge, cached there too if possible, and returned to the viewer.

Statistically, this all works out fine on cache hits, even though in cases like this one it introduces a round-trip disadvantage on cache misses and uncacheable content, because the Hong Kong edge must consult the Singapore regional edge, which must then send a request back to EC2 in Hong Kong. (Atlanta to Ohio to Hong Kong is admittedly more sensible that Hong Kong to Singapore to Hong Kong).

This behavior would almost certainly change in the future if CloudFront deploys a regional edge in Hong Kong.

Round-trip benchmarks for a single request also don't necessarily tell the entire performance story, particularly in low-traffic environments. CloudFront supports http/2 and thus multiple requests can be processing in parallel on a single browser connection, which CloudFront handles in parallel on the back-end via series of independent HTTP/1.1 requests that it makes to the origin. CloudFront handles TLS negotiation with the browser at the global edge, and reuses connections from the global edge to the regional cache as well as from the regional cache to the origin server, so on a site that is actively handling traffic it becomes more likely that requests benefit from these optimizations, reducing the actual round-trip times from what can be observed in isolation.

See How CloudFront Works with Regional Edge Caches and the list of global and regional edges.

See Origin Keep-Alive Timeout for discussion of a customization you can make to your distribution configuration to increase the amount of time CloudFront holds open back-end connections for reuse; the default value is only 5 seconds, but you can increase it to 60 seconds, or possibly more by special arrangement. See also Custom Origins/Persistent Connections.

Michael - sqlbot
  • 22,658
  • 2
  • 63
  • 86
  • Thank you very much. It helps me in planning the location of my origin servers. In this case, I presume, I should relocate my origin server to Singapore. However, when I do that, I do notice an increase in latency from US West Coast versus having it in HK (from 220 ms, it goes to about 400 ms). However, it's just US West Coast, and no such effect is seen in case of US Central or US East Coast. In fact, US East coast does clearly better than other US regions when the origin is in Singapore instead of HK. – Prashant D Oct 10 '19 at 05:30
  • Btw, is there a list of such regional caches? I'm wondering if there's one in India, or whether Indian content (that doesn't exist on the edge location) is checked with the Singapore hub and then it's fetched from Singapore if not present. – Prashant D Oct 10 '19 at 05:33
  • Do you have any idea whether there's also a 'third layer' that is based purely on the location of the origin server? In my case, for example, why would a request front-ended by Tokyo edge location be re-routed via Singapore? I would assume that Tokyo being a major hub would have a 'regional cache'? If it has a regional cache, shouldn't it be -- check in Japan, if not, send directly to HK? Why is it getting sent to SGP? EDIT -- found the LIST OF REGIONAL CACHES -- https://aws.amazon.com/about-aws/whats-new/2016/11/announcing-regional-edge-caches-for-amazon-cloudfront/ – Prashant D Oct 10 '19 at 05:38
  • As far as determining the outer (global) edge handling a request, note that there is a response header, `X-Amz-Cf-Pop` containing a value like `IAD89-C2`. The "IAD89" identifies the physical location of the CloudFront edge using Amazon's convention of numbering its buildings based on the nearest major airport, so IAD => Dulles International Airport, VA, US => Ashburn, VA, US. The "89" doesn't mean anything relevant to CloudFront since all amazon.com buildings (e.g. retail warehouses) also follow this numbering convention. The "-Cx" (if present) means "Cluster #x" in this specific building. – Michael - sqlbot Oct 10 '19 at 12:38
  • You found the list of regional edges from the original announcement when CloudFront rolled out the two-tier architecture. A more current list of both global and regional edges is linked in the answer, above, and can be found [here](https://aws.amazon.com/cloudfront/features/). – Michael - sqlbot Oct 10 '19 at 12:38
  • I also find this tool useful to get a rough idea concerning the latency between different regions: https://www.cloudping.co/grid – Marces Engel Jul 06 '23 at 08:49