2

I just migrated our production website over to Google App Engine last night, and now the app starts giving a 504 Gateway Timeout after about 10-15 minutes of inactivity. But I've been running a test version of the new site on GAE, in a different project, for months and it's never had this problem. I don't understand what is going on.

Below is the GAE dashboard graph which shows that pretty much all requests were returning as 5XX errors until 7am. I wasn't sure what was happening, so I just decided to re-deploy the app to see if that would kick things into working again... and for some reason, it seemed to do so.

enter image description here

After I re-deployed, the site started responding again. I clicked around for a while, fixed a bug, re-deployed... everything was good.

Then at 7:20, all of the requests turned into 504 Gateway Timeout errors again.

I saw it at 7:40, re-deployed without any changes and it's back up and working again.

What is going on?

p.s. I know that a dynamic gae instance will go idle and require time to spin back up, but that's not what is happening. I see that delay with my dev project gae, but then the page comes up after about 10 seconds. On my production gae, my browser just spins for a full minute and then gives me a 504 error and it keeps doing it until I re-deploy.

enter image description here

Kenny Wyland
  • 200
  • 1
  • 12
  • How did you solve this issue? were you using Flex o Std? Any insight or solution to this? – Joss Baron Aug 05 '20 at 15:35
  • 1
    @JossBaron This was a year ago and so my memory on it is weak, BUT... if I'm remembering the correct thing, it had to do with sending too many logging messages. As weird as that sounds, if I did too many `syslog()` or `error_log()` calls, the app would just kill itself. The frontend gateway would try to talk to the backend instance, but the instance was dead, so the gateway would eventually timeout. – Kenny Wyland Aug 25 '20 at 22:04

2 Answers2

1

According to OP:

He had too many syslog() or error_log() calls in too little time and this caused a backend issue where the app would crash and the frontend gateway would try to talk to the backend instance, but the instance was dead, so the gateway would eventually timeout.

The fix lies within spacing out the calls to log information.

Hipster Cat
  • 131
  • 5
0

This could be because your app scale down to 0 instances, this causes that new request experienced Cold boot to avoid this issue you can configure warmup request in order to improve the responsiveness

For example:

automatic_scaling:
  target_cpu_utilization: 0.68
  min_instances: 1
  max_instances: 10
  min_idle_instances:1

With this configuration you can guarantee at least 1 instance ready for incoming requests and avoiding to scale down to 0.

Usually when a deployment is ready 1 or 2 instances are spined up but without traffic an application can scale down to 0.

Jan Hernandez
  • 223
  • 1
  • 7