API Gateway cares about my Authorization header when it shouldn't

Question

I created a private REST API in API Gateway (with Lambda proxy integration), which needs to be accessible from a VPC. I've setup a VPC Endpoint for API Gateway in the VPC. The API is accessible from within the VPC, as expected.

The VPC endpoint (and indeed the entire VPC environment) is created via CloudFormation.

The API needs to consume an Authorization header, which is not something I can change. The content of that header is something specific to our company, it's not something standard. The problem is that when I add an Authorization header to the request, API Gateway rejects it with the following error (from API Gateway logs in CloudWatch):

IncompleteSignatureException
Authorization header requires 'Credential' parameter.
Authorization header requires 'Signature' parameter.
Authorization header requires 'SignedHeaders' parameter.
Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header.
Authorization=[the header content here]

If I remove the Authorization header, the request is accepted and I get the expected response from my lambda. The method I'm calling has Auth set to NONE.

The strange thing is that if I delete the VPC endpoint and create it manually via the console, it works correctly - the Authorization header is passed through to my lambda, instead of API Gateway inspecting and rejecting it.

I've torn the endpoint down and recreated it multiple times manually and with CloudFormation and the results are consistent. But I've compared them to each other and they look exactly the same: same settings, same subnets, same security groups, same policy. Since I can see no difference between them, I'm at a bit of a loss as to why it doesn't work with the CloudFormation version.

The only difference I've been able to find is in the aws headers for each version (with Authorization header removed, otherwise it doesn't get as far as logging the headers with the CF endpoint). With the CF endpoint, the headers include x-amzn-vpce-config=0 and x-amzn-vpce-policy-url=MQ==. With the manual endpoint I get x-amzn-vpce-config=1, and the policy-url header isn't included.

I've also tried changing the API to both set and remove the VPC endpoint (it can be set on the API in the Settings section), and redeployed it, but in either case it has no effect - requests continue to work/get rejected as before.

Does anyone have any ideas? I've posted this on the AWS forum as well, but just in case anyone here has come across this before...

If it's of any interest, the endpoint is created like so ([] = redacted):

ApiGatewayVPCEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    PrivateDnsEnabled: true
    PolicyDocument:
      Statement:
        - Action: '*'
          Effect: Allow
          Resource: '*'
          Principal: '*'
    ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
    SecurityGroupIds:
      - !Ref [my sec group]
    SubnetIds:
      - !Ref [subnet a]
      - !Ref [subnet b]
      - !Ref [subnet c]
    VpcEndpointType: Interface
    VpcId: !Ref [my vpc]

Perhaps you should consider using a custom authorizer instead of implementing the authorization logic into the lambda function. — kgiannakakis, Oct 17 '21 at 21:22
@kgiannakakis That's not possible. This service is a front for other services which make use of the Authorization header; this service itself shouldn't care about it, it just passes it through. There is zero scope to modify this, it's just the way it is and it needs to work like that, and it can work like that, we do this all the time with public apis, and it works with a private api when the endpoint is created manually, so I know this architecture works, there is just something somewhere going wrong... — 404, Oct 18 '21 at 08:38

404 · Accepted Answer · 2021-11-29T13:01:39.220

I've managed to get it working, and it's the most ridiculous thing.

This is the endpoint policy in CF (including property name to show it in context):

PolicyDocument:
  Statement:
    - Action: '*'
      Effect: Allow
      Resource: '*'
      Principal: '*'

This is how that policy appears in the console:

{
    "Statement": [
        {
            "Action": "*",
            "Effect": "Allow",
            "Resource": "*",
            "Principal": "*"
        }
    ]
}

This is how the policy appears in describe-vpc-endpoints:

"PolicyDocument": "{\"Statement\":[{\"Action\":\"*\",\"Resource\":\"*\",\"Effect\":\"Allow\",\"Principal\":\"*\"}]}"

Now let's look at the policy of a manually created endpoint.

Console:

{
    "Statement": [
        {
            "Action": "*",
            "Effect": "Allow",
            "Resource": "*",
            "Principal": "*"
        }
    ]
}

describe-vpc-endpoints:

"PolicyDocument": "{\n  \"Statement\": [\n    {\n      \"Action\": \"*\", \n      \"Effect\": \"Allow\", \n      \"Principal\": \"*\", \n      \"Resource\": \"*\"\n    }\n  ]\n}"

The console shows them exactly the same, and the JSON itself returned in describe-vpc-endpoints is the same except for some "prettifying" newlines and whitespace, surely that could have no effect whatsoever? Wrong! It's those newlines that make the policy actually work!

Anyway, the solution is to supply the policy as JSON, for example:

ApiGatewayVPCEndpoint:
  Type: AWS::EC2::VPCEndpoint
  Properties:
    PrivateDnsEnabled: true
    PolicyDocument: '
    {
      "Statement": [
        {
          "Action": "*",
          "Effect": "Allow",
          "Resource": "*",
          "Principal": "*"
        }
      ]
    }'
    ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
    SecurityGroupIds:
      - !Ref [my sec group]
    SubnetIds:
      - !Ref [subnet a]
      - !Ref [subnet b]
      - !Ref [subnet c]
    VpcEndpointType: Interface
    VpcId: !Ref [my vpc]

You can even put all the JSON on a single line, it will get the newline characters put in there by AWS at some point. It's just YAML that gets transformed to JSON without newlines and causes this issue.

With the CF resource like that, API Gateway accepts my Authorization header and passes it through to the Lambda without any issues.

API Gateway cares about my Authorization header when it shouldn't

1 Answers1