Why is Google App Engine throwing access forbidden errors?

Question

Could really use some help here. I have a GAE NodeJS app in the standard environment. Until a few days ago (09/23) it was running just fine, it would respond to requests as expected, etc.

Today, the app responds with 403's when I try to make any request to my appspot url. I'm 100% certain this is not a code issue, as if I deploy the same code to GAE in another project, it works fine. Furthermore, the only firewall rule is a wildcard to allow all traffic.

Edit: adding the only relevant-looking log entry I see from the project:

  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "status": {},
      "authenticationInfo": {
        "principalEmail": "address@domain.com"
      },
      "requestMetadata": {
        "callerIp": "x.x.x.x",
        "requestAttributes": {
          "time": "2021-09-23T15:04:05.198927Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "appengine.googleapis.com",
      "methodName": "google.appengine.v1.Services.UpdateService",
      "authorizationInfo": [
        {
          "resource": "apps/my-google-cloud-project-id/services/default",
          "permission": "appengine.services.update",
          "granted": true,
          "resourceAttributes": {}
        }
      ],
      "resourceName": "apps/my-google-cloud-project-id/services/default",
      "serviceData": {
        "@type": "type.googleapis.com/google.appengine.v1.AuditData",
        "updateService": {
          "request": {
            "name": "apps/my-google-cloud-project-id/services/default",
            "service": {
              "networkSettings": {
                "ingressTrafficAllowed": "INGRESS_TRAFFIC_ALLOWED_INTERNAL_AND_LB"
              }
            },
            "updateMask": "networkSettings"
          }
        }
      },
      "resourceLocation": {
        "currentLocations": [
          "us-east1"
        ]
      }
    },
    "insertId": "an-id",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-google-cloud-project-id",
        "zone": "",
        "module_id": "default",
        "version_id": ""
      }
    },
    "timestamp": "2021-09-23T15:04:05.131761Z",
    "severity": "NOTICE",
    "logName": "projects/my-google-cloud-project-id/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "some-operation-uuid",
      "producer": "appengine.googleapis.com/admin",
      "first": true
    },
    "receiveTimestamp": "2021-09-23T15:04:05.495890906Z"
  }

I don't recall making this change, and I'm not sure what the ingressTrafficAllowed value was before.

My suggestion is to check your App Engine's service account and see if something changed. Another thing is did you enabled IAP on your service? — Donnald Cucharo, Sep 29 '21 at 05:20
@Dondi as far as I can see, nothing changed with the service account. The IAP API was enabled but it itself was disabled. I disabled the API entirely on the project for good measure, but no change. — JustADude, Sep 29 '21 at 13:48
@yedpodtrzitko added the only relevant-seeming log entry I found to the original post. — JustADude, Sep 29 '21 at 13:53
Can you please check if the [following thread](https://serverfault.com/questions/1022496/app-engine-flexible-deployment-issue-403-resource-error) helps you? — Rajeev Tirumalasetty, Sep 29 '21 at 22:01

score 1 · Answer 1 · answered Sep 30 '21 at 07:15

1

Somehow the ingress setting on the GAE service got changed. I believe that issue was fixed by going to GCP console > App Engine > Services > select affected service(s) -> Edit ingress setting from the top, and select the appropriate value.

I say I believe this fixed the issue as I was still getting 403's on my appspot url after doing this, and ultimately I ended up deleting and re-creating the project from scratch, which got everything working again. Clearly there was some misconfiguration somewhere in my project, but GCP does not make it easy to diagnose what the issue might be.

answered Sep 30 '21 at 07:15

JustADude

123
9

I'm happy you got the problem solved, although the solution is quite unsatisfactory. – yedpodtrzitko Sep 30 '21 at 07:48
@yedpodtrzitko thank you. And 100% agree -- GCP was so obtuse about what was happening here, it's making me reconsider using it at all, and switching to something simpler and/or with better support options. Thankfully this was in dev, but were something like this to happen in prod... – JustADude Sep 30 '21 at 18:47

Why is Google App Engine throwing access forbidden errors?

1 Answers1