python-magic on GCP App Engine - failing to deploy

Question

I am working on Python app that will be deployed to GCP App Engine.
I tried to use python-magic functionality.
I've added import magic to my code and python-magic to the requirements.txt file.
But when I try to deploy code to App Engine it fails with the following error:

[2020-02-03 15:35:23 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/env/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/env/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/env/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/home/vmagent/app/main.py", line 11, in <module>
    import magic
  File "/env/lib/python3.6/site-packages/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation
[2020-02-03 15:35:23 +0000] [8] [INFO] Worker exiting (pid: 8)
[2020-02-03 15:35:23 +0000] [1] [INFO] Shutting down: Master
[2020-02-03 15:35:23 +0000] [1] [INFO] Reason: Worker failed to boot.

What I am missing to setup to make this work?

EDIT: sharing minimal setup to reproduce this error

main.py

# -*- coding: utf-8 -*-

import json
import base64
import magic

import logging
logging.basicConfig(level=logging.DEBUG)

from flask import Flask, request, abort

app = Flask(__name__)

@app.route('/parse', methods=['POST'])
def parse():

    try:
        #input should be a b64 encoded byte string
        input = request.get_json()['input']
    except Exception as e:
        abort(400, 'An error occured while reading in parameters: {}'.format(str(e)))

    decoded_input_data = base64.decodebytes(input)
    mime = magic.Magic(mime=True)
    file_ext = mime.from_buffer(decoded_input_data)

    returnData = {}
    returnData['Status'] = 200
    returnData['file_ext'] = file_ext

    return json.dumps(returnData), 200

@app.errorhandler(Exception)
def error(e):
    logging.exception(str(e))
    return json.dumps({'Status': e.code,
                       'Message': e.description}), e.code

if __name__ == '__main__':
    # This is used when running locally. Gunicorn is used to run the
    # application on Google App Engine. See entrypoint in app.yaml.
    app.run(host='127.0.0.1', port=8080, debug=True)

requirements.txt

Flask==1.0.2
gunicorn==19.9.0
google-cloud-vision==0.38.0
google-cloud-bigquery==1.11.2
google-cloud-storage==1.15.0
requests-toolbelt==0.9.1
protobuf==3.6.0
python-magic==0.4.15

app.yaml

runtime: python
env: flex
service: myservicename
entrypoint: gunicorn -b :$PORT main:app --timeout 240 --limit-request-line 0

runtime_config:
  python_version: 3

Daniel Ocando · Answer 1 · 2020-02-07T15:22:04.793

If you want to use the python-magic library you can follow the next procedure:

App Engine Standard

Use the venv module to create an isolated Python environment in a directory external to your project and activate it with:

python3 -m venv env
source env/bin/activate

Install the python-magic library with pip by running:

pip install python-magic

Add the library to your requirements.txt file. For example, if you are running the Quickstart from the Official documentation your requirements.txt file should look like this:

Flask==1.1.1
python-magic==0.4.15

Make the import of the library on the relevant module of your application's code. For example, if you are running the Quickstart from the Official documentation the relevant section of your main.py file should look like this:

...
# [START gae_python37_app]
from flask import Flask
import magic

# If `entrypoint` is not defined in app.yaml, App Engine will look for an app
# called `app` in `main.py`.
app = Flask(__name__)
...

App Engine Flexible The libmagic C library is not pre-installed on flex environment (here is the list of the installed libraries). You'll therefore need to create a custom runtime.

Changing your app.yaml to the following:

runtime: custom
env: flex

And adding the corresponding Dockerfile to your project's directory. For example:

FROM python:3.7
WORKDIR /app
COPY . /app
RUN apt-get update &&\
    apt-get install -y libmagic-dev
RUN pip install -r requirements.txt
EXPOSE 8080
CMD ["gunicorn", "main:app", "-b", ":8080", "--timeout", "300"]

should suffice to deploy the application successfully.

Notice that the Dockerfile simply installs the libmagic C library, afterwards installs your requirements.txt and finally runs gunicorn on the port required for App Engine. But feel free to adapt it to your needs.

I have done exactly this steps and I am receiving this error. — Michał Herman, Feb 06 '20 at 07:25
Notice that I didn't add the whole code of the main.py of the Quickstart. Just the section that imports the library. Would you mind sharing a minimum, reproducible, example of your code (including app.yaml, requirements.txt and relevant modules where you use the library)? — Daniel Ocando, Feb 06 '20 at 08:17
I have edited the question with whole minimal setup to reproduce this error — Michał Herman, Feb 06 '20 at 14:45
I edited the file an include a section for App Engine Flexible. As my first answer was using App Engine Standard. — Daniel Ocando, Feb 07 '20 at 15:22

python-magic on GCP App Engine - failing to deploy

1 Answers1