Google App Engine Flexible deploy throwing ERROR: (gcloud.app.deploy) HttpError accessing

Question

Hi I am getting this error when deploying nodejs application to flexible engine. I am unable to figure out where the issue is happening.

The error message I am getting

ERROR: (gcloud.app.deploy) HttpError accessing <https://appengine.googleapis.com/v1/apps/project-id/services/testws/versions?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 26 Mar 2021 20:37:27 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'transfer-encoding': 'chunked', 'status': '500', 'content-length': '109', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.

app.yaml

runtime: nodejs
env: flex
service: testws
manual_scaling:
  instances: 1
network:
  session_affinity: true
resources:
  cpu: 1
  memory_gb: 1
  disk_size_gb: 2
env_variables:
  KEY: "dev"
  PRIVATE_KEY_URL: "key"
skip_files:
  - ^node_modules/.*$

Gcloud SDK version

Google Cloud SDK 333.0.0
bq 2.0.65
core 2021.03.19
gsutil 4.60
kubectl 1.17.17

I have added the debug logs by running gcloud app deploy --verbosity="debug"

debug logs

>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.
Traceback (most recent call last):
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/Users/shashi/google-cloud-sdk/lib/surface/app/deploy.py", line 130, in Run
    use_legacy_apis=args.use_legacy_apis)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 678, in RunDeploy
    ignore_file=args.ignore_file)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 455, in Deploy
    extra_config_settings)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 172, in DeployService
    extra_config_settings)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 249, in _CreateVersion
    return self.client.apps_services_versions.Create(create_request)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/third_party/apis/appengine/v1/appengine_v1_client.py", line 830, in Create
    config, request, global_params=global_params)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 729, in _RunMethod
    http, http_request, **opts)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 350, in MakeRequest
    check_response_func=check_response_func)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 406, in _MakeRequestNoRetry
    check_response_func(response)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis.py", line 267, in _CheckResponseForApiEnablement
    http_wrapper.CheckResponse(response)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 223, in CheckResponse
    raise exceptions.BadStatusCodeError.FromResponse(response)
apitools.base.py.exceptions.BadStatusCodeError: HttpError accessing <https://appengine.googleapis.com/v1/apps/project-id/services/awsockets/versions?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Sun, 28 Mar 2021 22:29:25 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'transfer-encoding': 'chunked', 'status': '500', 'content-length': '109', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
>

Are you sure your application runs locally without any issues? Can you deploy the [quickstart for NodeJS](https://cloud.google.com/appengine/docs/standard/nodejs/quickstart) with no issues in your project? If the 500 is an actual issue with the Google infrastructure just try to deploy at a later time. If that's not the case check the Activity Logs for your project to check at which step of the underlaying process of issuing the Compute Engine instances for your App the deployment failed. — Daniel Ocando, Mar 28 '21 at 11:34
Hi Daniel, Yes I have tested it locally and it works flawlessly. I uninstalled and resinstalled gcloud sdk but the issue is same. — Shashi Kiran, Mar 28 '21 at 22:34
I got the logs from debug logs. I have updated it in the question — Shashi Kiran, Mar 28 '21 at 22:35
Are you able to deploy any other application? Such as the quickstart mentioned earlier? What about your network? Are you using some sort of proxy, or firewall that might be presenting issues? — Daniel Ocando, Mar 29 '21 at 05:26
I tried to deploy quickstart and got the same error. I am using home wifi. no VPN or Proxy. I can deploy other cloud run and use cloud build applications. only app engine is failing. — Shashi Kiran, Mar 29 '21 at 18:46
I would recommend you to open a support ticket with GCP if that's the case. The issue seems related to your specific project. Feel free to post an answer if you get it working! — Daniel Ocando, Mar 30 '21 at 06:22

Shashi Kiran · Accepted Answer · 2021-05-08T23:39:40.730

I found the reason for the error and the solution to fix it if anyone is facing this issue.

Reason - The App engine Flexible Service Account was accidentally deleted from the google cloud project. As mentioned in this link - service-account

The App Engine flexible environment service agent has the App Engine Flexible Environment Service Agent role. The role includes a set of permissions needed by Node.js flexible environment to manage your flexible environment apps. For example, this role includes permissions to perform the following tasks:

 1. Deploying a new version.
 2. Stopping or deleting existing versions.
 3. Automatic weekly restarts and system updates.

Solution - Add a service account by going to IAM and adding an account of name mentioned below with role App Engine Flexible Environment Service Agent. As mentioned in this link - restore-service-account

service-[YOUR_PROJECT_NUMBER]@gae-api-prod.google.com.iam.gserviceaccount.com

I have raised an issue with google tracker to change the error message. You can track the issue here - issue-tracker

Update - adding the below image for further clarity. if you try to add the deleted account, it will be suggested by the system.

i get an error `service-[YOUR_PROJECT_NUMBER]@gae-api-prod.google.com.iam.gserviceaccount.com` this email Email addresses and domains must be associated with an active Google Account, Google Workspace account, or Cloud Identity account. how did u get past this? — not_fubar_yet, May 07 '21 at 17:23
@not_fubar_yet I have added the screenshot for reference and updated my answer — Shashi Kiran, May 08 '21 at 23:40

score 1 · Answer 2 · answered May 26 '22 at 01:01

I had a similar error message when deploying to the standard environment. To resolve it, I added the verbosity=debug flag as in gcloud app deploy --verbosity="debug" as suggested.

Then I could see my earlier token replacement step was not working, leading to an invalid app.yaml

The verbose output shows the app.yaml like this.

2022-05-25T23:39:43.8269116Z DEBUG: Converted YAML to JSON: "{
2022-05-25T23:39:43.8269423Z   "basicScaling": {
2022-05-25T23:39:43.8269664Z     "idleTimeout": "900s",
2022-05-25T23:39:43.8269912Z     "maxInstances": 4
2022-05-25T23:39:43.8270102Z   },
2022-05-25T23:39:43.8270288Z   "entrypoint": {
2022-05-25T23:39:43.8270550Z     "shell": "npm run start:prod"
2022-05-25T23:39:43.8270764Z   },
2022-05-25T23:39:43.8271862Z   "instanceClass": "B4",
2022-05-25T23:39:43.8272089Z   "runtime": "nodejs16",
2022-05-25T23:39:43.8272316Z   "vpcAccessConnector": {
2022-05-25T23:39:43.8272882Z     "name": "projects/#{project_id}#/locations/#{region}#/connectors/sql-access-conn-#{environment}#"
2022-05-25T23:39:43.8273192Z   }
2022-05-25T23:39:43.8273380Z }"

Putting in a valid project_id for the vpcAccessConnector fixed the error.

Google App Engine Flexible deploy throwing ERROR: (gcloud.app.deploy) HttpError accessing

2 Answers2