0

I have a problem accessing a Google Cloud HTTP Function that is exposed through Firebase Hosting. I assume there is an issue with the CDN, because the function has not changed, but I do not see anything on the status pages:

Accessing the function directly, i.e. not through Firebase Hosting but using the https://{region}{app}.cloudfunctions.net/{name} URL seems to be running into the same issues.

Initially, I received this response after close to two minutes

content-type: text/plain;charset=US-ASCII
date: Tue, 06 Oct 2020 14:10:01 GMT
fastly-host: cache-hhn4049-HHN
server: Varnish
status: 503

Timed out while waiting on cache-hhn4049-HHN

This seems to indicate a problem with Fastly (listed as a subprocessor for Firebase), but their status page shows everything as operational.

After some time, I seem to be getting responses from a different party after a timeout of 15-30s:

(stripped HTML for readability)

accept-ranges: bytes
content-length: 323
content-type: text/html; charset=UTF-8
date: Tue, 06 Oct 2020 14:17:22 GMT
fastly-restarts: 1
server: Google Frontend
status: 500
vary: x-fh-requested-host, accept-encoding
x-cache: MISS
x-cache-hits: 0
x-cloud-trace-context: 234db5d36bf48ff0a385ed838e14676c
x-served-by: cache-hhn4049-HHN
x-timer: S1601993817.546227,VS0,VE25795

Error: Server Error
The server encountered an error and could not complete your request.
Please try again in 30 seconds.

Of course, waiting for 30 seconds does not help.

In the cloud functions log I do see some requests going through and taking between 400-500ms for normal execution. I can see one 'crash' finishing in 54ms, two longer executions at around 20s, and one timeout (60s). The function is used as a part of a custom authentication process, so there are usually a lot of requests.

Any advice on what I can do to understand what is the issue? Or a status page that can actually tell me where I can look for a problem?

Kariem
  • 4,398
  • 3
  • 44
  • 73
  • 1
    Contact Firebase support directly if you think there is a problem with the service. https://support.google.com/firebase/contact/support – Doug Stevenson Oct 06 '20 at 15:05
  • @DougStevenson Thank you for the suggestion. I have contacted support. I would like to know, if there is a different way to see what the problem is. The different status pages do not show any outages and it may take 2 or more business days for support to get back to me. Also, it would be interesting to know, if there is a different way to resolve the problem, as it seems there is no actual outage. If, however, there is an outage, I think it's helpful for other developers to find a solution. – Kariem Oct 06 '20 at 15:21
  • I received a response from Firebase Support that the teams is working on a solution and the [Google Cloud Status Page](https://status.cloud.google.com/) now shows _Google App Engine_ and _Google Cloud Support_ with a _Service Disruption_ annotation. In addition there is a message at the top: "_10% of Cloud Functions deployments are failing in us-central1_". This could be related to problems with deployment I also had when trying to resolve the issue, but I believe this does not address problems with the CDN. – Kariem Oct 06 '20 at 16:05

1 Answers1

3

firebaser here

There were indeed issues with deploying Cloud Functions in the past hour or so. We're working on adding the issue to the Firebase status page.

It should now mostly be back to normal. For the current status, see the yellow banner at the top of the Google Cloud dashboard. If there is no yellow banner, the problem is likely completely fixed.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
  • The Cloud Function had problems and I tried resolving them with a redeploy after about 45 minutes of debugging. Prior to that, I did not touch the function at all. – Kariem Oct 06 '20 at 16:12
  • Thanks Kariem. So you're saying that the initial problem started without a deploy? I saw two incident reports internally, but could figure out how the other one would surface to Cloud Functions. – Frank van Puffelen Oct 06 '20 at 16:53
  • Btw: the issue with **serving** of Cloud Functions responses should now also be solved as far as I can tell. – Frank van Puffelen Oct 06 '20 at 16:59
  • Yes, the initial problem started without a deploy. – Kariem Oct 12 '20 at 10:08