Catboost on Google AppEngine Python Flexible Environment

Question

I'm trying to set up a simple Google AppEngine app that would get a JSON data via POST request and apply Catboost classifier.

I'm using Python 3.6 and I've included catboost==0.8 in my requirements.txt file.

However, the deployment fails:

ERROR: (gcloud.app.deploy) Error Response: [9]
Application startup error:
[2018-04-24 16:42:04 +0000] [1] [INFO] Starting gunicorn 19.7.1
[2018-04-24 16:42:04 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2018-04-24 16:42:04 +0000] [1] [INFO] Using worker: sync
[2018-04-24 16:42:04 +0000] [9] [INFO] Booting worker with pid: 9
[2018-04-24 16:42:05 +0000] [9] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/lib/python3.6/site-packages/gunicorn/util.py", line 352, in import_app
    __import__(module)
  File "/home/vmagent/app/main.py", line 8, in <module>
    from catboost import CatBoostClassifier, Pool
  File "/env/lib/python3.6/site-packages/catboost/__init__.py", line 1, in <module>
    from .core import Pool, CatBoost, CatBoostClassifier, CatBoostRegressor, CatboostError, cv, train  # noqa
  File "/env/lib/python3.6/site-packages/catboost/core.py", line 50, in <module>
    _catboost = get_catboost_bin_module()
  File "/env/lib/python3.6/site-packages/catboost/core.py", line 46, in get_catboost_bin_module
    import _catboost
  ModuleNotFoundError: No module named '_catboost'

My base Dockerfile is the following:

FROM gcr.io/google-appengine/python

# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
RUN virtualenv /env -p python3

# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

RUN apt-get update; \
        apt-get install -y \
        build-essential \
        python-dev \
        python-setuptools \
        python-matplotlib \
        libatlas-dev \
        curl \
        ssh \
        libatlas3gf-base && \
        apt-get clean

# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

I suspect the problem is with sys.modules that doesn't register _catboost on importing catboost. On Google AppEngine Flexible Environment, Is there a way to use third-party apps that use C libraries, like Catboost?

Have you followed the other steps [from the docs](https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27)? — Phix, Apr 24 '18 at 17:00
@Phix those are the steps for Standard Environment. I'm using Flexible Environment. Not sure those steps are applicable. — Dmitri Lihhatsov, Apr 24 '18 at 17:01
In APP Engline Flex, the option available is simply to add the dependencies of your Library to a requirements.txt file in your application's source directory, as you have mentioned. The system is designed use pip to install any dependencies before starting your application. If this is not working on app engine, did it work [locally](https://cloud.google.com/appengine/docs/flexible/python/using-python-libraries#installing_dependencies_locally)? — oakinlaja, May 08 '18 at 00:00
@oakinlaja Locally everything works fine of course. Also, with Python 2.7 on Google App Engine all works fine. It's only because of Python 3.6 importing Catboost fails. And I think the problem is due to the fact that the Debian distro that the App Engine uses does not have libpython3.6 available. Only libpython3.4 for now. — Dmitri Lihhatsov, May 08 '18 at 08:55
There is no technical reason why AppEngine Flex would not support Python 3.6 + Catboost as it's entirely user configurable. The [documentation](https://cloud.google.com/appengine/docs/python/) confirms that the App Engine Flex supports the Python 3.6. This is because App Engine Flex allows you to ship your own container to Flex, or modify the [default container](https://cloud.google.com/appengine/docs/flexible/python/customizing-the-python-runtime). You can check this [link](https://github.com/Kaggle/docker-python/blob/master/Dockerfile#L341) on how to modify the default container — oakinlaja, May 14 '18 at 23:59

score 0 · Answer 1 · answered May 21 '18 at 23:33

There is no technical reason why AppEngine Flex would not support Python 3.6 + Catboost as it's entirely user configurable. The documentation confirms that the App Engine Flex supports the Python 3.6. This is because App Engine Flex allows you to ship your own container to Flex, or modify the default container. You can check this link on how to modify the default container

Catboost on Google AppEngine Python Flexible Environment

1 Answers1