Resize images on the fly in CloudFront and get them in the same URL instantly: AWS CloudFront -> S3 -> Lambda -> CloudFront

Question

TLDR: We have to trick CloudFront 307 redirect caching by creating new cache behavior for responses coming from our Lambda function.

You will not believe how close we are to achieve this. We have stucked so badly in the last step.

Business case:

Our application stores images in S3 and serves them with CloudFront in order to avoid any geographic slow downs around the globe. Now, we want to be really flexible with the design and to be able to request new image dimentions directly in the CouldFront URL! Each new image size will be created on demand and then stored in S3, so the second time it is requested it will be served really quickly as it will exist in S3 and also will be cached in CloudFront.

Lets say the user had uploaded the image chucknorris.jpg. Only the original image will be stored in S3 and wil be served on our page like this:

//xxxxx.cloudfront.net/chucknorris.jpg

We have calculated that we now need to display a thumbnail of 200x200 pixels. Therefore we put the image src to be in our template:

//xxxxx.cloudfront.net/chucknorris-200x200.jpg

When this new size is requested, the amazon web services have to provide it on the fly in the same bucket and with the requested key. This way the image will be directly loaded in the same URL of CloudFront.

I made an ugly drawing with the architecture overview and the workflow on how we are doing this in AWS:

Here is how Python Lambda ends:

return {
    'statusCode': '301',
    'headers': {'location': redirect_url},
    'body': ''
}

The problem:

If we make the Lambda function redirect to S3, it works like a charm. If we redirect to CloudFront, it goes into redirect loop because CloudFront caches 307 (as well as 301, 302 and 303). As soon as our Lambda function redirects to CloudFront, CloudFront calls the API Getaway URL instead of fetching the image from S3:

I would like to create new cache behavior in CloudFront's Behaviors settings tab. This behavior should not cache responses from Lambda or S3 (don't know what exactly is happening internally there), but should still cache any followed requests to this very same resized image. I am trying to set path pattern -\d+x\d+\..+$, add the ARN of the Lambda function in add "Lambda Function Association" and set Event Type Origin Response. Next to that, I am setting the "Default TTL" to 0.

But I cannot save the behavior due to some error:

Are we on the right way, or is the idea of this "Lambda Function Association" totally different?

Answer 1

Finally I was able to solve it. Although this is not really a structural solution, it does what we need.

First, thanks to the answer of Michael, I have used path patterns to match all media types. Second, the Cache Behavior page was a bit misleading to me: indeed the Lambda association is for Lambda@Edge, although I did not see this anywhere in all the tooltips of the cache behavior: all you see is just Lambda. This feature cannot help us as we do not want to extend our AWS service scope with Lambda@Edge just because of that particular problem.

Here is the solution approach:
I have defined multiple cache behaviors, one per media type that we support:

For each cache behavior I set the Default TTL to be 0.

And the most important part: In the Lambda function, I have added a Cache-Control header to the resized images when putting them in S3:

s3_resource.Bucket(BUCKET).put_object(Key=new_key, 
                                      Body=edited_image_obj,
                                      CacheControl='max-age=12312312',
                                      ContentType=content_type)

To validate that everything works, I see now that the new image dimention is served with the cache header in CloudFront:

Answer 2

You're on the right track... maybe... but there are at least two problems.

The "Lambda Function Association" that you're configuring here is called Lambda@Edge, and it's not yet available. The only users who can access it is users who have applied to be included in the limited preview. The "maximum allowed is 0" error means you are not a preview participant. I have not seen any announcements related to when this will be live for all accounts.

But even once it is available, it's not going to help you, here, in the way you seem to expect, because I don't believe an Origin Response trigger allows you to do anything to trigger CloudFront to try a different destination and follow the redirect. If you see documentation that contradicts this assertion, please bring it to my attention.

However... Lambda@Edge will be useful for setting Cache-Control: no-cache on the 307 so CloudFront won't cache it, but the redirect itself will still need to go all the way back to the browser.

Note also, Lambda@Edge only supports Node, not Python... so maybe this isn't even part of your plan, yet. I can't really tell, from the question.

Read about the Lambda@Edge limited preview.

The second problem:

I am trying to set path pattern -\d+x\d+\..+$

You can't do that. Path patterns are string matches supporting * wildcards. They are not regular expressions. You might get away with /*-*x*.jpg, though, since multiple wildcards appear to be supported.