python script starting 2 daemons

Question

My plan is to provide a script just as the title states. I've got an idea which I'll descibe below. If you think something sounds bad/stupid, I'd be grateful for any constructive comments, improvements, etc.

There are 2 services I want to start as daemons. One is required (a caching service), one is optional (http access to the caching service). I use argparse module to get --port to get caching service port and optional --http-port to get http access. I already have this and it works. Now I'd like to start the daemons. THe services are based on twisted, so they have to start the reactor loop. So far I would like to have two different processes: one for the service and second one for http access (though I know it might be done in a single async process).

Since starting twisted service is done via reactor loop (which is python code, not a shell script, since I don't use twistd yet), I think that using os.fork is better than subprocess (which would need a command line command to start the process). I can use os.fork to start daemons and touch service.pid and http.pid files, but I don't know how to access the child pid, since os.fork returns 0 for the child.

So the chld PID is what I'm missing. Moreover, if anything seems illogical or overcomplicated, please comment on that.

My current code looks like this:

#!/usr/bin/python
import argparse
import os

from twisted.internet import reactor

parser = argparse.ArgumentParser(description='Run PyCached server.')
parser.add_argument('port', metavar='port', type=int,
    help='PyCached service port')
parser.add_argument('--http-port', metavar='http-port', type=int, default=None,
    help='PyCached http access port')
args = parser.parse_args()

def dumpPid(name):
    f = open(name + '.pid', 'w')
    f.write(str(os.getpid()))
    f.flush()
    f.close()

def erasePid(name):
    os.remove(name + '.pid')

def run(name, port, factory):
    dumpPid(name)
    print "Starting PyCached %s on port %d" % (name, port)
    reactor.listenTCP(port, factory)
    reactor.run()
    erasePid(name)
    print "Successfully stopped PyCached %s" % (name,)

# start service (required)
fork_pid = os.fork()
if fork_pid == 0:
    from server.service import PyCachedFactory
    run('service', args.port, PyCachedFactory())
else:
    # start http access (optional)
    if args.http_port:
        fork_pid = os.fork()
        if fork_pid == 0:
            from server.http import PyCachedSite
            addr = ('localhost', args.port)
            run('http', args.http_port, PyCachedSite(addr))
        else:
            pass

I run it with:

./run.py 8001 # with main service only

or:

./run.py 8001 --http-port 8002 # with additional http

System shutdown is done via single shell script:

#!/bin/bash

function close {
    f="$1.pid"
    if [ -f "$f" ]
    then
        kill -s SIGTERM `cat "$f"`
    fi    
}

close http
close service

Answer 1

Since starting twisted service is done via reactor loop (which is python code, not a shell script, since I don't use twistd yet), I think that using os.fork is better than subprocess (which would need a command line command to start the process).

You should use twistd. If not, then you should write a Python script for launching the daemon. Then you should use the subprocess module (or reactor.spawnProcess) to launch the child process.

Using os.fork without immediately proceeding to one of the os.exec* functions is broken. A large amount of state is shared between the parent and child created by os.fork. You can't be sure that this sharing won't break something (and I can tell you it will break some things in Twisted).

Here are some links to discussions of fork-without-exec issues that might help you get more of an idea of what a troublesome area this is.

• Twisted epoll reactor issues - https://twistedmatrix.com/pipermail/twisted-python/2013-October/027611.html
• stdlib ssl security issues - https://mail.python.org/pipermail/python-dev/2013-October/129834.html
• is twisted incompatible with multiprocessing events and queues?
• multiprocessing memory usage and twisted/gevents