Proper way to start thread from python daemon

Question

I need to write simple daemon with web interface.

The idea is to use python-daemon package and run wsgiref.simple_server inside one thread.

Daemon works fine with the following code :

import daemon
import logging
import time
import signal
import threading

logfilename = '/var/log/testdaemon.log'
logger = logging.getLogger("DaemonLog")
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
    '%(asctime)s:%(levelname)s:%(message)s',
    '%Y-%m-%d %H:%M:%S')
handler = logging.FileHandler(logfilename)
handler.setFormatter(formatter)
logger.addHandler(handler)

def initial_program_setup():
    logger.info('daemon started')

def do_main_program():
    while True:
        time.sleep(1)
        logger.info('another second passed')

def program_cleanup(signum, frame):
    logger.info('daemon stops')
    context.terminate(signum, frame)

def reload_program_config(signum, frame):
    logger.info('reloading config')

context = daemon.DaemonContext()

context.signal_map = {
    signal.SIGTERM: program_cleanup,
    signal.SIGHUP: 'terminate',
    signal.SIGUSR1: reload_program_config,
    }

context.files_preserve = [handler.stream]

initial_program_setup()

with context:
    do_main_program()

But if I start a thread in initial_program_setup() like this :

def web_gui():
    logger.info('weg gui started')

web = threading.Thread(target=web_gui)
web.setDaemon(True)

def initial_program_setup():
    logger.info('daemon started')
    web.start()

then looks like daemon exits after thread completes. Adding something like

while True:
    time.sleep(1)

to web_gui() (to make thread run forever, like a web server should) makes it even worse: even the line web gui started doesn't show up in log.

My questions are:

1. Why this doesn't work? What's the proper way to start thread in daemon?
2. Maybe there is a better way to control daemon through web interface? With such architecture, I think I should start new thread for each interface page, which is hard to scale.

Thanks.

Answer 1

This is a limitation (discussion thread starts here) of the daemon library.

TL;DR: Your options are:

• Don't use daemon, because it is unrepairably broken in this regard.
• Start the thread within the "with daemoncontext" block.

Long version:

When the daemon library switches to daemon context, it does a double-fork. That means it first forks and then kills the parent process. A new fork does not have any thread, so exiting the parent process equates killing your webgui thread. Ultimately, any solution to this problem must start any permanent threads in the newly created child process. The downside of doing this in the daemon context is that you are no longer able to propagate potential errors to the user. Ideally you'd double-fork, but do not exit the parent process, then set up your daemon and right before entering the main loop make the parent process exit. This is not achievable with the daemon library or any library implementing the PEP3143 draft.

Answer 2

Worked for me. You basically have to imagine that setting up the daemon context is like a frontal lobotomy :-)

Here is what I figured is easy to follow My main only parses arguments and, if flagged to run in the background, creates the daemon context. Thereafter, it executes a common main loop function _main_common(), which is where everything is done including all threads being created:

args = parse_arguments(args)
if args.foreground:
   # we run in the foreground, so just run the main loop
   _main_common()
else:
   # we run in the background, so now create the daemon context and run the main loop
   context = create_daemon_context(context, detach_process=True)
   with context:
      _main_common()