Error with App Engine split health checks

Question

I am unable to deploy my App Engine Flex Python project with split health checks.

I have enabled health checks:

$ gcloud beta app describe
...
featureSettings:
  splitHealthChecks: true

Using the docs as a model, my app.yaml has:

readiness_check:
  path: '/health_check'
  check_interval_sec: 5
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2
  app_start_timeout_sec: 600

liveness_check:
  path: '/health_check'
  check_interval_sec: 30
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2

But when I deploy, it fails due to the following error:

  File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 109, in init_process
    super(ThreadWorker, self).init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/local/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
    __import__(module)
  File "/env/local/lib/python2.7/site-packages/vmruntime/wsgi.py", line 48, in <module>
    wsgi_config.get_module_config_filename())
  File "/env/local/lib/python2.7/site-packages/vmruntime/wsgi_config.py", line 48, in get_module_config
    return appinfo_includes.Parse(f)
  File "/env/local/lib/python2.7/site-packages/google/appengine/api/appinfo_includes.py", line 57, in Parse
    appyaml, _ = ParseAndReturnIncludePaths(appinfo_file, open_fn)
  File "/env/local/lib/python2.7/site-packages/google/appengine/api/appinfo_includes.py", line 82, in ParseAndReturnIncludePaths
    appyaml = appinfo.LoadSingleAppInfo(appinfo_file)
  File "/env/local/lib/python2.7/site-packages/google/appengine/api/appinfo.py", line 2417, in LoadSingleAppInfo
    listener.Parse(app_info)
  File "/env/local/lib/python2.7/site-packages/google/appengine/api/yaml_listener.py", line 227, in Parse
    self._HandleEvents(self._GenerateEventParameters(stream, loader_class))
  File "/env/local/lib/python2.7/site-packages/google/appengine/api/yaml_listener.py", line 178, in _HandleEvents
    raise yaml_errors.EventError(e, event_object)
EventError: Unexpected attribute 'failure_threshold' for object of type ReadinessCheck.

I'm following the documentation so I'm not sure why it would be triggering an error here.

The only thing I can imagine is that my 'site-packages' is perhaps out of date? I don't believe my Docker should install any google/appengine/ libraries, but I'm at a loss for what I could be doing wrong.

Do you have the latest SDK? Run `gcloud components update` and try again? — BrettJ, Dec 23 '17 at 21:10
If you run `gcloud version` what version of `app-engine-python` is shown? — BrettJ, Dec 24 '17 at 15:43
Looks like `app-engine-python` 1.9.64. Seems 1.9.65 was released a few days ago, but `gcloud components update` shows no updates still. — Mike Lambert, Dec 24 '17 at 21:13
Oh, is this something that should be pulled in by the Docker image? I'm using the deprecated `gcr.io/google_appengine/python-compat-multicore`, so perhaps this is fallout from that? Let me poke around a bit more, regenerate some images, and get back to you. — Mike Lambert, Dec 24 '17 at 21:14
Rebuilt all my docker images (based off `python-compat-multicore`), and still seeing problems. Where are the updated appengine python files included from...`gcr.io/google_appengine/python`? If so, I guess that means I'd need to rebuild a version of `python-compat-multicore` for myself, to make this work? — Mike Lambert, Dec 25 '17 at 06:28
Probably should post your entire `app.yaml` in your question. There are a few other settings that might be at play here. — BrettJ, Dec 29 '17 at 23:16
Full app.yaml: https://github.com/mikelambert/dancedeets-monorepo/blob/master/server/app.yaml (only difference, being that these healthcheck lines are commented-out in this linked version, so that my push will work) — Mike Lambert, Dec 30 '17 at 06:25
Hey @brettj, The python-compat-runtime has a git commit 23 Jul 2017, but the docker image was never rebuilt to include it, and is the source of my problems. Is it possible to rebuild the deprecated runtime? — Mike Lambert, Mar 04 '18 at 19:07

score 1 · Accepted Answer · answered Mar 05 '18 at 05:31

So seems the above failure scenario is caused by the gcr.io/google_appengine/python-compat-multicore docker image being out of date.

The github is up to date with the latest python files, but the associated Docker image was never rebuilt.

I have fixed this particular error by checking out the github and rebuilding the docker image myself, and pointing my appengine flex stuff at my own parent-built-image.

(I now have a separate problem, with split healthchecks being sent to the default paths, and it not letting me override the paths. I have filed that as bug in the Google cloud issue tracker. But at least the above question is now solved for me)

This worked! I can't tell you how many hours I wasted going down rabbit holes here. Amazing, thank you! To get this working for me, I also had to set the undocumented `host` field in my health checks, e.g., `host: mysite.com` — speedplane, Oct 18 '19 at 14:12

Error with App Engine split health checks

1 Answers1