AWS CloudFront signed cookies + S3 works in REST client app, but not browser

Question

I'm making this question as detailed as possible so that anyone else who runs into the same issues will have a comprehensive resource for figuring this out and getting it working.

The Goal

The goal is to use signed cookies so that an authenticated user in my application can access any of their files freely, without having to sign URLs.

The S3 and CloudFront Config

I'm pretty sure most of this is correct, but just for the sake of providing a complete picture, I'll include the setup I have.

S3 Config

I have a bucket we'll call my-storage. It has the following CORS configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

The bucket policy is:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xxx"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-storage/*"
        },
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-storage/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": "http://localhost:8080/*"
                }
            }
        }
    ]
}

The localhost exception is just so I can build/test my app locally, since HTTP cookies wouldn't work properly due to cross-domain issues.

CloudFront Distribution

I have a CloudFront distribution that uses this bucket as the origin. For the sake of this post, we'll say the CNAME is files.mysite.com. This is the origin configuration:

The behavior config is a bit much to screenshot and post, but the important details are:

Allowed HTTP Methods: GET, HEAD, OPTIONS
Cache and origin request settings: Use a cache policy and origin request policy
Cache Policy: Managed-CachingOptimized
Origin Request Policy: Managed-CORS-S3Origin
Restrict Viewer Access: Yes
Trusted Signers: Self

Positive Test Results in Insomnia (REST client)

I'm using the Insomnia REST client to test this out, to remove the browser from the equation, and it looks like it works ok.

I make a request to my API to return signed cookies, which I can see in the response header:

date: Tue, 25 Aug 2020 15:09:35 GMT
x-amzn-requestid: xxx
access-control-allow-origin: https://web.mysite.com
set-cookie: CloudFront-Policy=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure
set-cookie: CloudFront-Key-Pair-Id=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure
set-cookie: CloudFront-Signature=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure
x-amz-apigw-id: xxx
vary: Origin
x-powered-by: Express
x-amzn-trace-id: Root=xxx;Sampled=1
access-control-allow-credentials: true
x-cache: Miss from cloudfront
via: 1.1 xxx.cloudfront.net (CloudFront)
x-amz-cf-pop: ORD52-C1
x-amz-cf-id: xxx

And Insomnia stores the cookies in the client. Then I make a GET request for a file https://files.mysite.com/users/xx/xx.mp3. I get a 200 response and the binary data of the file, no problem. The headers show me the cookies were sent properly:

> GET /users/9dbb70d7-3d17-4215-8966-49815e461dee/audio/d76bb13d-0e1d-45dc-b7e5-9cb8fb6dee1a/workfile.mp3 HTTP/1.1
> Host: files.mysite.com
> User-Agent: insomnia/2020.3.3
> Cookie: CloudFront-Key-Pair-Id=xxx; CloudFront-Signature=xxx; CloudFront-Policy=xxx
> Origin: https://web.mysite.com
> Accept: */*

Great! So in theory, this should work.

Actual Browser Result

Here's what happens in the web app though. I authenticate, and I see the API request go out to get the signed cookies:

GET https://api.mysite.com/private/get-signed-cookie

{
  "Response Headers (1.373 KB)": {
    "headers": [
      {
        "name": "access-control-allow-credentials",
        "value": "true"
      },
      {
        "name": "access-control-allow-origin",
        "value": "https://web.mysite.com"
      },
      {
        "name": "date",
        "value": "Tue, 25 Aug 2020 15:16:28 GMT"
      },
      {
        "name": "set-cookie",
        "value": "CloudFront-Policy=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure"
      },
      {
        "name": "set-cookie",
        "value": "CloudFront-Key-Pair-Id=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure"
      },
      {
        "name": "set-cookie",
        "value": "CloudFront-Signature=xxx; Domain=mysite.com; Path=/users/; HttpOnly; Secure"
      },
      {
        "name": "vary",
        "value": "Origin"
      },
      {
        "name": "via",
        "value": "1.1 xxx.cloudfront.net (CloudFront)"
      },
      {
        "name": "x-amz-apigw-id",
        "value": "xxx"
      },
      {
        "name": "x-amz-cf-id",
        "value": "xxx"
      },
      {
        "name": "x-amz-cf-pop",
        "value": "ORD52-C1"
      },
      {
        "name": "x-amzn-requestid",
        "value": "xxx"
      },
      {
        "name": "x-amzn-trace-id",
        "value": "xxx"
      },
      {
        "name": "x-cache",
        "value": "Miss from cloudfront"
      },
      {
        "name": "X-Firefox-Spdy",
        "value": "h2"
      },
      {
        "name": "x-powered-by",
        "value": "Express"
      }
    ]
  }
}

At this point, it's worth noting that I cannot see the cookies in Firefox Dev Tools! I can only assume they didn't get stored.

And when the browser tries to access something via the CloudFront distribution:

GET https://files.mysite.com/users/9dbb70d7-3d17-4215-8966-49815e461dee/audio/d76bb13d-0e1d-45dc-b7e5-9cb8fb6dee1a/workfile.mp3

I get a 403 Forbidden response with this body:

<?xml version="1.0" encoding="UTF-8"?><Error><Code>MissingKey</Code><Message>Missing Key-Pair-Id query parameter or cookie value</Message></Error>

And sure enough, the request headers show no sign of Cookie being sent:

{
  "Request Headers (535 B)": {
    "headers": [
      {
        "name": "Accept",
        "value": "audio/webm,audio/ogg,audio/wav,audio/*;q=0.9,application/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5"
      },
      {
        "name": "Accept-Encoding",
        "value": "gzip, deflate, br"
      },
      {
        "name": "Accept-Language",
        "value": "en-US,en;q=0.5"
      },
      {
        "name": "Connection",
        "value": "keep-alive"
      },
      {
        "name": "Host",
        "value": "files.mysite.com"
      },
      {
        "name": "Origin",
        "value": "https://web.mysite.com"
      },
      {
        "name": "Range",
        "value": "bytes=0-"
      },
      {
        "name": "Referer",
        "value": "https://web.mysite.com/dashboard"
      },
      {
        "name": "User-Agent",
        "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko/20100101 Firefox/79.0"
      }
    ]
  }
}

What am I missing here? All the URLs talking to each other have the same domain (the API that issues the cookies, the web client, the CloudFront distribution). The Express API has the right CORS config, I'm pretty sure:

router.use(
  cors({
    origin(origin, callback) {
      if (/\.mysite\.com$/.test(origin)) {
        callback(null, true);
      } else {
        callback(new Error('Not allowed by CORS'));
      }
    },
    credentials: true,
  }),
);

I'm totally stumped. Any help on this would be greatly appreciated!

score 5 · Accepted Answer · answered Aug 25 '20 at 18:16

Ok, so there were a lot of steps to fix this. I might not have them all down exactly, but here's what I changed/fixed:

Client-Side Issues

The client was using axios to fetch a binary file, and an HTMLAudioElement to fetch an audio file. Both had to be updated to send credentials (cookies). For the axios call, that's axios.get(url, { withCredentials: true }) and for the HTMLAudioElement it's audioEl.crossOrigin = 'use-credentials'. Now both of these requests will send HTTP cookies to CloudFront.

API Gateway Changes

Some CORS stuff had to be set up here. It's too deep for this post, but I'm using the Serverless Framework to configure API Gateway which makes things a lot easier! In the serverless.yml file I had to make some changes:

          cors:
            origins:
              - https://localhost:8080
              - https://*.mysite.com
            headers:
              - Content-Type
              - X-Amz-Date
              - Authorization
              - X-Api-Key
              - X-Amz-Security-Token
              - X-Amz-User-Agent
              - x-user-id # this is my own header I pass from the web client
            allowCredentials: true

This basically sets up the API Gateway pre-flight OPTION endpoint so it returns the proper CORS headers. It creates this smart script in API Gateway to handle multiple origins:

#set($origin = $input.params("Origin"))
#if($origin == "") #set($origin = $input.params("origin")) #end
#if($origin.matches("https://localhost:8080") || $origin.matches("https://.+[.]mysite[.]com")) #set($context.responseOverride.header.Access-Control-Allow-Origin = $origin) #end

Again, too deep to get into in this answer. But anyone working with serverless should check out the Serverless Framework. It's a huge time-saver!

S3 Config

S3's CORS configuration had to be modified. The single change to the above config is:

<AllowedOrigin>https://*.mysite.com</AllowedOrigin>

The over-arching rule here across the whole stack is that if you're using Access-Control-Allow-Credentials: true, you can't have the allowed origin be *.

AWS CloudFront signed cookies + S3 works in REST client app, but not browser

The Goal

The S3 and CloudFront Config

S3 Config

CloudFront Distribution

Positive Test Results in Insomnia (REST client)

Actual Browser Result

1 Answers1

Client-Side Issues

API Gateway Changes

S3 Config

Linked