1

As of this morning all of my build pipelines have started to fail. No code in many of the pipelines have changed, and none of the cloudbuild.yaml files have changed.

I've got a few NodeJS applications that have a CloudBuild.yaml file that looks similar to this:

steps:

- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args: ['-c', 'gcloud config set app/cloud_build_timeout 1600']

#Install Dependencies 
- name: node
  entrypoint: npm
  args: ['install']

#Build an optimized version of the website
- name: node
  entrypoint: npm
  args: ['run', 'build']
  env:
  - [LIST OF ENV VARIABLES HERE -- Omitted for the question"

#Deploy to AppEngine using the Branch as the version
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args: ['-c', 'gcloud app deploy $_ENV_YAML']

timeout: '1600s'

options:
  machineType: 'N1_HIGHCPU_8'

Since this morning CloudBuild seems to get stuck after "Uploading Files" and eventually errors out:

    Step #3: Beginning deployment of service [development]...
Step #3: ╔════════════════════════════════════════════════════════════╗
Step #3: ╠═ Uploading 7 files to Google Cloud Storage                ═╣
Step #3: ╚════════════════════════════════════════════════════════════╝
Step #3: File upload done.
Step #3: Updating service [development]...
Step #3: ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................failed.
Step #3: ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred.

When I turn on debugging:

..............................DEBUG: Operation [apps/munch-web/operations/8048d88b-1993-4155-ad6f-9322f88e44f3] complete. Result: {
    "done": true,
    "error": {
        "code": 13,
        "message": "An internal error occurred."
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "endTime": "2021-07-02T19:10:58.770Z",
        "insertTime": "2021-07-02T18:55:55.020Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/munch-web/services/development/versions/20210702t185547",
        "user": "498370915597@cloudbuild.gserviceaccount.com"
    },
    "name": "apps/munch-web/operations/8048d88b-1993-4155-ad6f-9322f88e44f3"
}
failed.
DEBUG: (gcloud.app.deploy) Error Response: [13] An internal error occurred.
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/usr/lib/google-cloud-sdk/lib/surface/app/deploy.py", line 130, in Run
    use_legacy_apis=args.use_legacy_apis)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 691, in RunDeploy
    service_account=service_account)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 463, in Deploy
    extra_config_settings, service_account)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 213, in DeployService
    poller=done_poller)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 314, in WaitForOperation
    sleep_ms=retry_interval)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 264, in WaitFor
    sleep_ms, _StatusUpdate)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 326, in PollUntilDone
    sleep_ms=sleep_ms)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 243, in RetryOnResult
    if not should_retry(result, state):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 183, in IsDone
    encoding.MessageToPyValue(operation.error)))
googlecloudsdk.api_lib.app.operations_util.OperationError: Error Response: [13] An internal error occurred.
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred.

I can't seem to pinpoint where the issue is here. The crazy part is when I run this pipeline locally on my machine it deploys just fine with no issues to AppEngine.

Any idea where I can start troubleshooting?

Crash
  • 11
  • 3
  • 1
    Maybe a different gcloud version? Try to get the current one on Cloud Build, and try to update your local installation with the same (or a close version). It's might be a change on the CLI – guillaume blaquiere Jul 02 '21 at 21:21
  • I tried pulling the last two versions and still had the same issue. That was my exact thought too, but that didn't help :/ – Crash Jul 03 '21 at 00:07
  • it should be an internal bug of Google... Let's try next week! – guillaume blaquiere Jul 03 '21 at 18:02
  • Hi OP. I suggest using the same `gcloud` version on your build config, similar to what you're using where the deployment is working. Since this is an internal error, we don't have access to what's going on to your project so I suggest filing a case to [support](https://cloud.google.com/support-hub). – Donnald Cucharo Jul 05 '21 at 01:08

1 Answers1

0

It happened exactly at this time for us ! we checked lots of thing but finally emailed to google cloud support service and it was them bug not us! and they fixed it.

Misa Qarahqani
  • 1