Amazon CloudFront CORS specific delay

Question

I am experiencing a delay between the serving of CORS requests but direct requests are served fine. I am using this to distribute media streams via HTTP so it is very important to reduce the startup delay.

There is approximately 90-180 seconds between when a media manifest is available via the CloudFront distribution (via a direct request from the browser or curl) and when the CORS requests from the player on our website return success. I have enabled OPTIONS request forwarding in the CloudFront distribution and have included the results of an OPTIONS request too. I have included the result of a curl request and the corresponding result from the network tab from Chrome Dev tools below. Note that these requests were made from the same client within 15 seconds of each other (the curl request was sent first).

=== CURL request ===

*   Trying 54.192.135.101...
* Connected to <exampleDistributionID>.cloudfront.net (1.1.1.1) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*    subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.cloudfront.net
*    start date: Sep 17 00:00:00 2015 GMT
*    expire date: Dec 15 23:59:59 2016 GMT
*    subjectAltName: <exampleDistributionID>.cloudfront.net matched
*    issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 Secure Server CA - G4
*    SSL certificate verify ok.
> GET /path/to/manifest/stream.m3u8 HTTP/1.1
> Host: <exampleDistributionID>.cloudfront.net
> User-Agent: curl/7.47.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/vnd.apple.mpegurl
< Content-Length: 1435
< Connection: keep-alive
< Server: nginx/1.9.10
< Date: Sun, 17 Apr 2016 00:26:06 GMT
< Last-Modified: Sun, 17 Apr 2016 00:26:05 GMT
< ETag: "5712d81d-59b"
< Cache-Control: no-cache
< Access-Control-Allow-Origin: *
< Accept-Ranges: bytes
< X-Cache: Miss from cloudfront
< Via: 1.1 f687c6e8ce478528ab87681ac35779ab.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: P01_dDWZRWZ0lzAqROqOMnaipstK484vPWnicw3F0kcG_7elxBGNkQ== 

<...Content of stream.m3u8...>

===Chrome Request===

Screenshot of the Chrome Dev tools network tab showing the 404 error received

===OPTIONS Request===

*   Trying 1.1.1.1...
* Connected to <exampleDistributionID>.cloudfront.net (1.1.1.1) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*    subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.cloudfront.net
*    start date: Sep 17 00:00:00 2015 GMT
*    expire date: Dec 15 23:59:59 2016 GMT
*    subjectAltName: <exampleDistributionID>.cloudfront.net matched
*    issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 Secure Server CA - G4
*    SSL certificate verify ok.
> OPTIONS /path/to/manifest/stream.m3u8 HTTP/1.1
> Host: <exampleDistributionID>.cloudfront.net
> User-Agent: curl/7.47.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Content-Length: 0
< Connection: keep-alive
< Server: nginx/1.9.10
< Date: Sun, 17 Apr 2016 22:05:15 GMT
< Access-Control-Allow-Origin: http://my.origin.com
< Access-Control-Allow-Methods: GET, OPTIONS
< Access-Control-Allow-Headers: Authorization
< Access-Control-Allow-Credentials: true
< X-Cache: Miss from cloudfront
< Via: 1.1 ed2825b48bb51b4febd93a82e71f7ed9.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: WY-KPfTlNTenTjWyYF9GS4ikyrGMQONAm4mXpbuKpHzfBk_xKfxG2w==
< 
* Connection #0 to host <exampleDistributionID>.cloudfront.net left intact

At a loss to see the error in my config, any help would be greatly appreciated.

Answer 1

CloudFront tries to protect the origin server from unnecessary requests for unavailable objects by caching error responses for 5 minutes, by default.

It seemed apparent from the question that the most likely explanation would be that the object was being requested (for whatever reason) before it actually existed, and the error response was being cached for a few minutes, leading to what appeared to be a "delay" in the object's availability. But CloudFront has no propagation delays, because CloudFront is a pull-through cache -- there's nothing to propagate.

Your curl test appears to succeed, but fails to actually prove anything, apparently because (among other potential reasons) you didn't include an Origin: header in your curl request. This makes the curl request semantically different than the one sent by the browser.

When evaluating whether an object is available to be served from the cache, CloudFront doesn't only consider the path. Most of the headers that are forwarded to the origin server are also compared, and if the headers to be forwarded with this request don't match the headers sent with a prior request with a cached response available, the cached response won't be used, and a request will instead be sent to the origin. Its response will be cached along with the headers that were sent.

So, the following two requests:

GET /object HTTP/1.1
Host: www.example.com

and

GET /object HTTP/1.1
Host: www.example.com
Origin: http://www.example.com

...are (assuming the Origin: header is being forwarded to the origin server, as it would need to be for CORS) treated as two different, essentially unrelated requests, by necessity -- CloudFront doesn't know whether the origin server might modify its responses based on the request headers sent. The responses to these two requests would be cached separately, and each of them would only served be up in response to future, matching requests.

If the distribution is configured to forward cookies or query strings, these are also stored along with the cached response, which will only be served up in response to requests that exactly match the original request that generated the cached response -- based on all of the forwarded parameters. (This is why unnecessarily forwarding information your origin server doesn't need will hurt your cache ratio.)

Setting the distribution's Error Caching Minimum TTL for 404 errors down to 0 resolves this issue, by preventing caching of the 404 responses.