Currently i'm deploying my Flask application to an Ubuntu server (AWS). When I try to start the uwsgi server and look with journalctl to the logs I notice a kind of warning/error.
Can I ignore it? I don't know how to solve it or where it came from. Stuck on this for 2 days right now. Who can help me?
Error:
*** Operational MODE: preforking ***
Jan 04 15:27:11 ip-172-31-39-12 uwsgi[21781]: unable to load configuration from from multiprocessing.semaphore_tracker import main;main(10)
In my case this error was due to using uWSGI 2.0.17.1 with Flask 1.0.2 and scikit-learn 0.20.0.
Internally, scikit-learn imports joblib which, at import time, tries to spawn a semaphore tracking process (sklearn/externals/joblib/_multiprocessing_helpers.py).
The semaphore tracking process is spawned by generating a command with the name of the current executable and appending "-c 'from multiprocessing.semaphore_tracker import main;main(fd)".
The name of the current executable is expected to be 'python' but this is not so when using uWSGI. The resulting command is "/usr/local/bin/uwsgi -c 'from multiprocessing.semaphore_tracker import main;main(fd)" which fails and outputs the above error message.
A workaround, as documented here is to set the environment variable JOBLIB_MULTIPROCESSING=0.
Note that the only consequence of this, in my situation, was to generate a defunct uWSGI process that was eventually cleaned up.
You are ettempting to spawn subprocess(es) from inside your application (or from a lib you are using). According to this, an additional co-process is being also spawned - the semaphore tracker, responsible for returning back to system all named semaphores your subprocesses create. This is kind of important task, because if the named semaphore is leaked (not deleted), the associated system resource is occupied until next reboot.
The system has limited number of those resources, because they are located in the shared memory. You can ignore this if you are sure, that the number of named semaphores used by your app is not significant.
Notice that every type of lock defined in the multiprocessing module is a named semaphore under the hood. Moreover, every instance of multiprocessing.Queue, Barier, etc. instantiate their own locks.
If for instance you are spawning many processes (workers) and each of them is instantiating multiprocessing.Lock or multiprocessing.RLock, the number of leaked (not deleted) named semaphores may be significant, exhausting the limit quickly, causing your app or others to run out of resources.
Here is a link to an explanation of those issues: https://docs.python.org/3/library/multiprocessing.html?highlight=semaphore%20tracker#contexts-and-start-methods
Hey I've been struggling with the same problem, and while I don't know how to actually stop the specific semaphore warning from popping up changing some of my uWSGI options has helped ameliorate the problem.
My .ini config file is below:
[uwsgi]
module = wsgi:app
master = true
processes = 16
socket = api.sock
chmod-socket = 660
vacuum = true
harakiri = 30
die-on-term = true
max-requests = 3
The additions I made are the "harakiri" and "max-request" options. The harakiri option means if a request takes longer than 30 seconds the worker will recycle itself, the max request option means after three requests the worker will recycle itself. It seems to be working, so my theory is that while the semaphores may be untracked they are tied in some way to the worker, and recycling them regularly improves performance.
This is a "dumb fight" of a memory leak and I wish I had a more elegant solution, but has been working for me the past few days. Good luck!
In my case, I solved the problem by upgrading the joblib pip package from 0.13.2 to 0.14.1. The error disappears after the upgrade.
