112

I have a cron job that runs every 10 minutes and updates the content-type and x-amz-meta. But since yesterday it seems like after the cron job run, Amazon is not picking up the changes made (refreshing his cache).

I even went and made the changes manually but no change...

When a video is uploaded it has a application/x-mp4 content-type and the cron job changes it to video/mp4.

Although S3 has the right content type video/mp4 cloudfront shows application/x-mp4(old content-type) ....

The cron job has been working for the last 6 months without a problem.

What is wrong with amazon caching? How can i synchronize the caching?

Rico
  • 58,485
  • 12
  • 111
  • 141
mamesaye
  • 2,033
  • 2
  • 32
  • 49

8 Answers8

84

Use Invalidations to clear the cache, you can put the path to the files you want to clear, or simply use wild cards to clear everything.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidating-objects-api

This can also be done using the API! http://docs.aws.amazon.com/cloudfront/latest/APIReference/API_CreateInvalidation.html

The AWS PHP SDK now has the methods but if you want to use something lighter check out this library: http://www.subchild.com/2010/09/17/amazon-cloudfront-php-invalidator/

user3305600's solution doesn't work as setting it to zero is the equivalent of Using the Origin Cache Headers.

Roy Shmuli
  • 4,979
  • 1
  • 24
  • 38
Neo
  • 11,078
  • 2
  • 68
  • 79
65

As to the actual code

get your CloudFront distribution id

aws cloudfront list-distributions

Invalidate all files in the distribution, so CloudFront fetches fresh ones

aws cloudfront create-invalidation --distribution-id=S11A16G5KZMEQD --paths /

My actual full release script is

#!/usr/bin/env bash

BUCKET=mysite.com
SOURCE_DIR=dist/

export AWS_ACCESS_KEY_ID=xxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxx
export AWS_DEFAULT_REGION=eu-west-1


echo "Building production"
if npm run build:prod ; then
   echo "Build Successful"
else
  echo "exiting.."
  exit 1
fi


echo "Removing all files on bucket"
aws s3 rm s3://${BUCKET} --recursive


echo "Attempting to upload site .."
echo "Command:  aws s3  sync $SOURCE_DIR s3://$BUCKET/"
aws s3  sync ${SOURCE_DIR} s3://${BUCKET}/
echo "S3 Upload complete"

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=S11A16G5KZMEQD --paths / --profile=myawsprofile

echo "Deployment complete"

References

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/get-invalidation.html

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/create-invalidation.html

Dr Manhattan
  • 13,537
  • 6
  • 45
  • 41
  • Thanks for posting this--I needed to invalidate a distribution to *fully* rid my app of an old, overwritten file version that kept popping up. – SexxLuthor May 18 '17 at 08:26
  • 9
    Instead of deleting all files in the bucket first, you can use the `--delete` option of `aws s3 sync` to do that for you. i.e. `aws s3 sync ${SOURCE_DIR} s3://${BUCKET}/ --delete` – Ryan McGeary May 18 '17 at 23:30
  • 3
    Worthwhile to mention here, at path use "/*" invalidate all files in bucket and "/folder_name/*" to invalidate files in folder name. also I think it costs money – Karan Jul 30 '18 at 12:12
  • Per the cloudfront pricing page as of today. "No additional charge for the first 1,000 paths requested for invalidation each month. Thereafter, $0.005 per path requested for invalidation." – Nick Dec 30 '20 at 22:26
  • 1
    beware that a path of `/` does not invalidate the entire distribution (I'm not exactly sure what it does do); you need to use `/*` to do that – Andy Jul 21 '21 at 12:59
  • As a followup, make sure to put the `/*` inside of quotes like this: `"/*"`. This makes the command: `aws cloudfront create-invalidation --distribution-id=[ID_HERE]--paths "/*"`. If you don't do that, your command might pick up a list of files or folders on the local machine as the invalidation list! Example after running that command on ubuntu within Github Actions without the quotes showed up like this in the AWS invalidations panel (object paths): `[/lib32, /home, /proc, ...]`. Not what you want! – Sam May 25 '23 at 20:49
49

Here is a manual way to invalidate the cache for all files on CloudFront via AWS

  1. Open your CloudFront Distributions list, and click the desired distribution ID (circled in red in screenshot below) you want to clear it's cache. enter image description here
  2. Click 'Invalidations' tab (see selected tab highlighted in blue in the screenshot below).
  3. Click 'Create invalidation' button (circled in red in the screenshot below) enter image description here
  4. Insert /* in the object paths input in order to clear cache of all files.
  5. Click 'Create invalidation' button. enter image description here
benshabatnoam
  • 7,161
  • 1
  • 31
  • 52
Luke Phillips
  • 3,372
  • 1
  • 11
  • 15
  • 1
    Would this be a temporary change? Would it I need to delete the invalidation? Could I delete it immediately, or would I need to wait a few hours? – 2540625 Oct 25 '21 at 02:05
  • 2
    It's a temporary change and you don't need to delete it afterwards. It runs immediately after you create it and clears the entire cache (if you use `/*`). Just saved my day! – Sebastiaan Luca Nov 18 '21 at 12:35
23

S3 is not used for real time development but if you really want to test your freshly deployed website use

http://yourdomain.com/index.html?v=2
http://yourdomain.com/init.js?v=2

Adding a version parameter in the end will stop using the cached version of the file and the browser will get a fresh copy of the file from the server bucket

Vishwanath
  • 6,284
  • 4
  • 38
  • 57
HimalayanCoder
  • 9,630
  • 6
  • 59
  • 60
12

Cloudfront will cache a file/object until the cache expiry. By default it is 24 hrs. If you have changed this to a large value, then it takes longer.

If you anytime needs to force clear the cache, use the invalidation. It is charged separately.

Another option is to change the URL (object key), so it fetches the new object always.

Dane O'Connor
  • 75,180
  • 37
  • 119
  • 173
Sony Kadavan
  • 3,982
  • 2
  • 19
  • 26
11

If you're looking for a minimal solution that invalidates the cache, this edited version of Dr Manhattan's solution should be sufficient. Note that I'm specifying the root / directory to indicate I want the whole site refreshed.

export AWS_ACCESS_KEY_ID=<Key>
export AWS_SECRET_ACCESS_KEY=<Secret>
export AWS_DEFAULT_REGION=eu-west-1

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=<distributionId> --paths / --profile=<awsprofile>

Region Codes can be found here

You'll also need to create a profile using the aws cli. Use the aws configure --profile option. Below is an example snippet from Amazon.

$ aws configure --profile user2
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: text
Francis
  • 1,798
  • 1
  • 26
  • 32
1

(edit: Does not work)As of 2014, You can clear your cache whenever you want, Please go thorough the Documentation or just go to your distribution settings>Behaviors>Edit

Object Caching Use (Origin Cache Headers) Customize

Minimum TTL = 0

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html

user3305600
  • 43
  • 3
-1

I believe using * invalidate the entire cache in the distribution. I am trying at the moment, I would update it further

invalidate request screenshot

Update:

It worked as expected. Please note that you can invalidate the object you would like by specifying the object path.

greengreyblue
  • 389
  • 3
  • 12
  • What does invalidate cache actually do? I have it in my pipeline but I always have to hard reload my website to see the changes. I use s3 sync to update website files on S3. I think invalidate cache only clears files from Cloudfront Edge location but not the browsers cache. – Haseeb Burki Jun 12 '19 at 07:11
  • Using * where and how? – 2540625 Oct 25 '21 at 02:02