os.process on apache2 loads in usr/sbin/ and does not recognise command (module)

Question

I am trying to run a subprocess command in my flask app hosted on an apache2 server.

When I run the subprocess(confirmed code works outside apache2):

@app.route("/ScanResult")
def scan_result():
    process = subprocess.run('face_recognition ./known_faces/ ./unknown_faces/', shell=True, check=True,
                             stdout=subprocess.PIPE, universal_newlines=True)
    output = process.stdout
    flash(output)
    return render_template("scanresult.html")

I recieve an error:

Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
 Command line: '/usr/sbin/apache2'

/bin/sh: 1: face_recognition: not found
ERROR:webApp:Exception on /ScanResult [GET], referer: http://faces-app.duckdns.org:2020/Scan
Traceback (most recent call last):, referer: http://faces-app.duckdns.org:2020/Scan
   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app, referer: http://faces-app.duckdns.org:2020/Scan
    response = self.full_dispatch_request(), referer: http://faces-app.duckdns.org:2020/Scan
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
    rv = self.handle_user_exception(e), referer: http://faces-app.duckdns.org:2020/Scan
 File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception, referer: http://faces-app.duckdns.org:2020/Scan
    reraise(exc_type, exc_value, tb), referer: http://faces-app.duckdns.org:2020/Scan
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise, referer: http://faces-app.duckdns.org:2020/Scan
    raise value, referer: http://faces-app.duckdns.org:2020/Scan
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
    rv = self.dispatch_request(), referer: http://faces-app.duckdns.org:2020/Scan
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
    return self.view_functions[rule.endpoint](**req.view_args), referer: http://faces-app.duckdns.org:2020/Scan
   File "/var/www/webApp/webApp/__init__.py", line 368, in scan_result, referer: http://faces-app.duckdns.org:2020/Scan
   stdout=subprocess.PIPE, universal_newlines=True), referer: http://faces-app.duckdns.org:2020/Scan
 File "/usr/lib/python3.6/subprocess.py", line 438, in run, referer: http://faces-app.duckdns.org:2020/Scan
 output=stdout, stderr=stderr), referer: http://faces-app.duckdns.org:2020/Scan
 subprocess.CalledProcessError: Command 'face_recognition .../known_faces/ .../unknown_faces/' returned non-zero exit status 127., referer: http://faces-app.duckdns.org:2020/Scan

The python modules are installed in venv and the flask app works as expected with the exception of this subprocess.

How do I launch the command terminal into the venv and not in usr/sbin/apache2 or /bin/sh:?

I do have the venv activated.

I have attempted to pass the env to the subprocess statement without success.

 process = subprocess.run('face_recognition ./known_faces/ ./unknown_faces/', shell=True, check=True,
                             stdout=subprocess.PIPE, universal_newlines=True,
                             env={"PATH":"/home/thomas/venv/apache_flask_venv/bin/"},)

This came from this answer. add env to subprocess

Following the addition of the path to the module location changed the error:

[Tue Sep 01 19:02:40.459001 2020] [mpm_event:notice] [pid 6552:tid 140697108794304] AH00491: caught SIGTERM, shutting down
[Tue Sep 01 19:02:40.622309 2020] [mpm_event:notice] [pid 6725:tid 140301125479360] AH00489: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Tue Sep 01 19:02:40.622638 2020] [core:notice] [pid 6725:tid 140301125479360] AH00094: Command line: '/usr/sbin/apache2'
Dlib was compiled to use SSE41 instructions, but these aren't available on your machine.
Dlib was compiled to use SSE41 instructions, but these aren't available on your machine.
Traceback (most recent call last):
  File "/home/thomas/venv/apache_flask_venv/bin/face_recognition", line 11, in <module>
    sys.exit(main())
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/click/core.py", line 760, in main
    _verify_python3_env()
  File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
    " mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.

This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8
[Tue Sep 01 19:02:54.732537 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789] ERROR:webApp:Exception on /ScanResult [GET], referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732679 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789] Traceback (most recent call last):, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732706 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732739 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     response = self.full_dispatch_request(), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732762 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732786 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     rv = self.handle_user_exception(e), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732809 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732832 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     reraise(exc_type, exc_value, tb), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732854 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732877 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     raise value, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732899 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.732993 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     rv = self.dispatch_request(), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733019 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/home/thomas/venv/apache_flask_venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733043 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     return self.view_functions[rule.endpoint](**req.view_args), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733066 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/var/www/webApp/webApp/__init__.py", line 369, in scan_result, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733089 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     stdout=subprocess.PIPE, universal_newlines=True,), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733111 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]   File "/usr/lib/python3.6/subprocess.py", line 438, in run, referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733133 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789]     output=stdout, stderr=stderr), referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733164 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789] subprocess.CalledProcessError: Command '/home/thomas/venv/apache_flask_venv/bin/face_recognition ../known_faces/ ../unknown_faces/' returned non-zero exit status 1., referer: http://faces-app.duckdns.org:2020/Scan
[Tue Sep 01 19:02:54.733206 2020] [wsgi:error] [pid 6729:tid 140300676757248] [client 185.194.96.195:57789] , referer: http://faces-app.duckdns.org:2020/Scan

I am starting to review the error it appears to be related to the click module. unix bytes not unicode issue

If i can run this in a terminal I should be able to run it via my flask app surely , no?

(apache_flask_venv) thomas@apacheflask:/var/www/webApp/webApp$ face_recognition ./faces/ ./scan/
Dlib was compiled to use SSE41 instructions, but these aren't available on your machine.
./scan/singleperson.jpg,singleperson

I have tried the suggested solution:

export LC_ALL = en_US.utf-8
export LANG = en_US.utf-8

Inserted at the beginning of the .py file. Causes error:

[Tue Sep 01 19:24:17.852576 2020] [wsgi:error] [pid 7472:tid 140654836156160] [client 185.194.96.195:59238]     export LC_ALL = en_US.utf-8
[Tue Sep 01 19:24:17.852667 2020] [wsgi:error] [pid 7472:tid 140654836156160] [client 185.194.96.195:59238]                 ^
[Tue Sep 01 19:24:17.852729 2020] [wsgi:error] [pid 7472:tid 140654836156160] [client 185.194.96.195:59238] SyntaxError: invalid syntax

How can I modify my sub.process to match the suggested solution:

Instead you need to do this:

input = 'Input here'
in_stream = io.BytesIO(input.encode('utf-8'))
sys.stdin = io.TextIOWrapper(in_stream, encoding='utf-8')
out_stream = io.BytesIO()
sys.stdout = io.TextIOWrapper(out_stream, encoding='utf-8')

This causes problems if the terminal is incorrectly set and Python does not figure out the encoding. In that case, the Unicode string will contain error bytes encoded as surrogate escapes.

Do I need to change my locale?

my locale:

(apache_flask_venv) thomas@apacheflask:/var/www/webApp/webApp$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Modified the locale to the suggested, without any change to error.

(apache_flask_venv) thomas@apacheflask:/var/www/webApp/webApp$ locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=C.UTF-8

Is face_recognition in /home/thomas/venv/apache_flask_venv/bin/? Does it work if you provide the full path, e.g. `/home/thomas/venv/apache_flask_venv/bin/face_recognition ./known_faces/ ./unknown_faces/`? — Josh Cooley, Sep 01 '20 at 18:47
face_recognition is there. I have added the 'new' error caused by include the path to the module. — ThomasATU, Sep 01 '20 at 19:05

score 0 · Accepted Answer · answered Sep 02 '20 at 19:46

The only solution I have found for this issue, so far -- Is to utilise python2 instead of python3 - which required installing the required dlib and face_recognition into python2 environment and then calling python2.6 before the command.

Instead of this, I have created a method that replicates the functionality of the command-line tool.

unknown_images = os.listdir("/var/www/Webapp/webApp/scan/")
    for unknown_image in unknown_images:
        loaded_image = face_recognition.load_image_file(unknown_image)
        query_faces = face_recognition.face_encodings(loaded_image)
        faces_found = len(query_faces)
        faces = []
        if faces_found:
            faces_dict = get_faces_dict("/var/www/Webapp/webApp/faces/")
            face_encodings = list(faces_dict.values())
            for query_face in query_faces:
                match_results = face_recognition.compare_faces(
                face_encodings, query_face)
                for idx, match in enumerate(match_results):
                    if match:
                        match = list(faces_dict.keys())[idx]
                        match_encoding = face_encodings[idx]
                        dist = face_recognition.face_distance([match_encoding], query_face)[0]
                        faces.append({
                        "Filename":str(unknown_image),
                        "Name": match,
                        "Accuracy": dist
        })

    flash("Scan Results: " + str(faces))

If any one has the complete solution to fixing the issue associated with locale variable, please post. Besides re-installing and building in the python2 environment, the suggested solution of modifiying the LANG etc doesn't not work.

os.process on apache2 loads in usr/sbin/ and does not recognise command (module)

1 Answers1