AWS API gateway returns 403 for some and 200 for others

Question

I have a apigateway endpoint that returns 200 for me, but when it's called by a third party they get 403.

I request via curl and python requests and get 200 for both

Bash:

curl -X POST -v --http1.1 https://939pd1ndql.execute-api.us-east-1.amazonaws.com/default/bitbucket-events

Python

requests.post('https://939pd1ndql.execute-api.us-east-1.amazonaws.com/default/bitbucket-events',

I get 200 response for each request.

However when a third party calls the endpoint they get

HTTPSConnectionPool(host='939pd1ndql.execute-api.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: /default/bitbucket-events (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 403 Forbidden',)))

The third part is bitbucket - I am trying to create bitbucket app (really just a JSON payload telling bitbucket to create a webhook):

I do not have control over how bitbucket performs the requests and the request is very opaque but I pointed it at ngrok and intercepted the request it makes:

POST /default/bitbucket-events HTTP/1.1
Host: 939pd1ndql.execute-api.us-east-1.amazonaws.com
User-Agent: python-requests/2.22.0
Content-Length: 2292
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: application/json
Sentry-Trace: 00-41043c2935294252aa25ac44716a2300-86324af91ef0493e-00
X-Forwarded-For: 104.192.142.247
X-Forwarded-Proto: https
X-Newrelic-Id: VwMGVVZSGwQJVFVXDwcPXg==
X-Newrelic-Transaction: PxQPB1daXQMHVwRWAQkDUQUIFB8EBw8RVU4aWl4JDVcDUgoEBVcLVlNXDkNKQQoBBlZRAAQHFTs=

{LOTS OF JSON HERE}

Nothing in the request that bitbucket sends looks like it could cause this problem.

The response I get from the curl command is:

*   Trying 3.84.56.177...
* TCP_NODELAY set
* Connected to 939pd1ndql.execute-api.us-east-1.amazonaws.com (3.84.56.177) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.execute-api.us-east-1.amazonaws.com
*  start date: Jul 22 00:00:00 2021 GMT
*  expire date: Aug 20 23:59:59 2022 GMT
*  subjectAltName: host "939pd1ndql.execute-api.us-east-1.amazonaws.com" matched cert's "*.execute-api.us-east-1.amazonaws.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> POST /default/bitbucket-events HTTP/1.1
> Host: 939pd1ndql.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 12 Apr 2022 22:00:39 GMT
< Content-Type: application/json
< Content-Length: 0
< Connection: keep-alive
< x-amzn-RequestId: 78585bb0-5db4-4273-9333-45ef8b44952d
< Access-Control-Allow-Origin: *
< x-amz-apigw-id: QfN1IHrSoAMFrMw=

I have now devolved the apigateway to be just a mock endpoint that returns 200 response:

and I have set the logging to be very loud:

But I only see log entries as a result of the curl and python request I make. The bitbucket request does not result in a log line.

Could this mean the bitbucket request is being rejected by AWS before my api gateway is handling the request? I have no WAF enabled

As you can tell I am running out of ideas.

you have `bitbucket-events`, it needs to be `default/bitbucket-events` — Ermiya Eskandary, Apr 16 '22 at 15:28
in the app descriptor? Here's the app descriptor: see it has /default/bitbucket-events https://s3.eu-west-2.amazonaws.com/codereview.doctor/bitbucket-staging.json — Rich Tier, Apr 16 '22 at 21:59

Register Sole · Accepted Answer · 2022-04-21T02:10:16.410

1

I replicated your setup, but with my own API Gateway. I was able to install the app though, so strongly suspect it is something to with your API Gateway setup.

I am using the exact same app descriptor, with only the URL being different.

{
    "key": "codereview.doctor.staging",
    "name": "Code Review Doctor Staging",
    "description": "Target lambdas with 'staging' version alias",
    "vendor": {
        "name": "Code Review Doctor",
        "url": "https://codereview.doctor"
    },
    "baseUrl": "https://fj7987nlx3.execute-api.ap-southeast-1.amazonaws.com",
    "authentication": {
        "type": "jwt"
    },
    "lifecycle": {
        "installed": "/default/bitbucket-events",
        "uninstalled": "/default/bitbucket-events"
    },
    "modules": {
        "webhooks": [
            {
                "event": "pullrequest:created",
                "url": "/default/bitbucket-events"
            },
            {
                "event": "pullrequest:updated",
                "url": "/default/bitbucket-events"
            },
            {
                "event": "pullrequest:fulfilled",
                "url": "/default/bitbucket-events"
            }
        ]
    },
    "scopes": ["account", "repository", "pullrequest"],
    "contexts": ["account"]
}

My API GW POST configuration looks exactly like yours, so the difference may be somewhere else.

Note that I have deleted my API GW stage, so you will not be able to test using mine for now.

edited Apr 21 '22 at 02:10

answered Apr 20 '22 at 03:11

Register Sole

3,206
1
14
22

AWS API Gateway has concept of "stages" - think of them like versioned endpoints v1, v2, v3 etc. The arbitrary stage name I picked was "default", so the bitbucket-events endpoint is available at default/bitbucket-events. Note for debugging purposes I use ngrok and logged the actual request being made by bitbucket. It was indeed requesting /default/bitbucket-events ... I then manually replayed the request to the API gateway and got 200 response. Also I set up a simple proxy via https://beeceptor.com/console/coderviewdoctor and bitbucket worked fine: returned 200. so strange! – Rich Tier Apr 20 '22 at 13:49
@DjangoDoctor you're right sorry forgot about the stage being in the URL. I replicated your setup and able to install in BitBucket though. The only difference is using my API GW mock. So strongly suspect something in your API GW setup. Edited the answer. – Register Sole Apr 21 '22 at 02:04
@DjangoDoctor To find the difference, can you share more of your API GW config? I have covered as much as I can with the info currently provided. – Register Sole Apr 21 '22 at 02:15
wow thanks a lot the for the help. I have done states > default > export > Export as Swagger + API Gateway Extensions > JSON and got this: https://gist.github.com/code-review-doctor/2f8a2c4ac48841dd2fc2ccf2142c60d8 If you export yours too we can maybe see a difference – Rich Tier Apr 21 '22 at 12:49
Alright I think it is Open API instead of Swagger. I've also made it as similar as possible, but there are few differences left. [My config](https://gist.github.com/WinsonT/ffcf4665e2f3ff3faef73252463e80e4). Initially, I didn't have the `x-amazon-apigateway-gateway-responses` and the `Access-Control-Allow-Origin` in POST and GET. But I added them to make it similar, re-deployed, and strangely, it still works for me. Also have left my API online for now. – Register Sole Apr 22 '22 at 03:28
thank. I was able to install yours. I see mine is us-east and yours is ap-southeast-1 - I will try making an AWS gateway in a different region too. – Rich Tier Apr 22 '22 at 08:53
1

indeed, when I changed the region to us-east-1 to us-east-2 (keeping everything exectly the same) the problem went away. Bitbucket hates us-east-1! – Rich Tier Apr 30 '22 at 22:21
your input helped me discover this so marking your as answer – Rich Tier Apr 30 '22 at 22:21
1

@RichTier there is indeed something wrong with calling apis in us-east-1. I've enabled all possible tracing, and it didn't show anything; Mock server receives requests with no issue. Hence, something infront of API Gateway rejects the requests (judging by headers some internal Cloudfront or WAF). Using another region with no config or code modifications...fixed it – Vladyslav Didenko Apr 03 '23 at 22:05

score 0 · Answer 2 · answered Apr 03 '23 at 22:03

I found few possible reasons mentioned in AWS docs & one that is not. Probably it saved me 10-20 hours of troubleshooting

Forbidden -> Something wrong with Authorisation
Forbidden -> Wrong Stage name/not deployed
3rd party tools have issues somehow connecting to "us-east-1" API Gateway endpoint. Deploying in another region issue solves it immediately. Haven't found any possible reasons. Postman ok, curl ok, 3rd party libs not ok (not even log traces; seems to cut of in flight)

AWS API gateway returns 403 for some and 200 for others

2 Answers2