My ultimate goal: I want a single Python process that spawns a Flask app, a Celery worker, a Redis server, and maybe eventually a Webpack server. So far so good. I also want to be able to intercept every line that those processes log and prepend a color-coded label so it's easy to separate in the terminal what output came from what process. This is where I am really struggling.
Here is a simplified version of the approach that is closest to working (adapted from this SO answer):
def start_thread(func):
x = threading.Thread(target=func)
x.start()
def execute(cmd):
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line:
break
print("test:", line.rstrip())
def start_celery():
cwd = os.getcwd()
os.chdir('..') # eventually this will take an argument to figure out which dir it needs to change to
execute(['celery', '-A', '<my_module>.run_celery:celery', 'worker', '--pool=gevent',
'--concurrency=500', '--loglevel', 'info'])
os.chdir(cwd)
def start_redis():
execute(['redis-server'])
def start_flask():
execute(['flask', 'run'])
if __name__ == '__main__':
# parse a bunch of args
start_functions = {'celery': start_celery, 'redis': start_redis, 'flask': start_flask}
[start_thread(func) for arg, func in start_functions.items() if getattr(args, arg)]
Another approach to needing to change directories, using shell=True
as an argument to subprocess.Popen
and passing 'cd ..; celery -A cookiecutter_mbam.run_celery:celery worker --pool=gevent '--concurrency=500 --loglevel info'
seems to be working equivalently.
This correctly prints "test:" (well, in fact "test: b'") in front of each line of Flask and Redis output. It does not print anything in front of Celery output. Also, some, but not all, of the Celery output I am used to seeing when a worker starts is suppressed.
I do see:
[2019-12-28 10:17:00,563: INFO/MainProcess] mingle: searching for neighbors
[2019-12-28 10:17:01,628: INFO/MainProcess] mingle: all alone
[2019-12-28 10:17:01,650: INFO/MainProcess] celery@Katies-MacBook-Air.local ready.
[2019-12-28 10:17:01,665: INFO/MainProcess] pidbox: Connected to redis://localhost:6379//.
But normally I would also see a blue ascii Celery logo, some configuration information, and list of registered Celery functions, and that's gone. On the other hand, if I change start_celery
to simply:
def start_celery():
subprocess.run(['celery', '-A', '<my_module>.run_celery:celery', 'worker', '--pool=gevent', '--concurrency=500', '--loglevel', 'info'])
and execute the file from within the directory where Celery is initialized I see all my Celery output.
What causes this? Is this output from Celery printed to my screen by some method other than having been fed to standard out? Is that why I don't see it?