AppEngine Flexible instances constantly respawning

Question

I am deploying a Go application using AppEngine flexible. Below is my app.yaml. Sometimes after I deploy it stabilizes at 1 instance (it's a very low load application), but most of the time it constantly respawns upwards of 6 instances. My logs are filled with messages showing the new instances being created. There is nearly zero load on this application, why is AppEngine flexible constantly destroying and respawning instances?

Log showing constant respawning:

app.yaml

runtime: go
api_version: go1
env: flex

handlers:
- url: /.*
  script: _go_app

health_check:
  enable_health_check: True
  check_interval_sec: 10
  timeout_sec: 4
  unhealthy_threshold: 2
  healthy_threshold: 2

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 10
  cool_down_period_sec: 120 # default value
  cpu_utilization:
    target_utilization: 0.5

What happens if you send a get request to url '/_ah/health' of any one of your instance? — Anthony Kong, Jan 11 '17 at 08:39
This may be an issue with the platform. We should first rule out that the instance is actually unhealthy. Respawns are most often caused by failing or unresponsive health checks. Per your settings, an instance need only be unresponsive for 20 seconds (2 health checks) to potentially cause a respawn (3 to be safe). Do your health check logs `/_ah/health` show any failures or responses further than 30 seconds apart? What's the timeline of this respawn issue? What is the CPU and memory usage like for instances of your application? Does a Hello World **go** flex application do this? — Nicholas, Jan 12 '17 at 14:41
The nginx.health_check log shows no failed health checks. This often occurs after a new deployment, but then stops after a few hours. There is constant instance respawning for a few hours, then it eventually stabilizes at 2 instances. I have not tested with a Hello World golang flex application. — Kevin Flanagan, Jan 12 '17 at 15:21
I tried various settings of the timeout_sec, unhealthy_threshold and cool_down_period_sec variables, all with the same results. Significant instance respawning for hours, then eventual settling. — Kevin Flanagan, Jan 13 '17 at 01:43

score 1 · Accepted Answer · answered Jan 14 '17 at 23:03

The problem was with my health check function. It originally looked like this:

func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
    return
}

I then discovered this sentence in the documentation on how instances are managed:

You can write your own custom health-checking code. It should reply to /_ah/health requests with a HTTP status code 200. The response must include a message body, however, the value of the body is ignored (it can be empty).

So I changed the health check function to write a simple "ok" in response:

func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("ok"))
    return
}

The instances now behave according to my autoscale settings! The respawning is gone.

I obviously should have read the documentation closer, but there was zero indication of a problem in the health check logs. All health checks looked like they were passing. Hopefully this info is helpful to others.

AppEngine Flexible instances constantly respawning

1 Answers1