1

This is my code:

from multiprocessing import Pool, Lock
from datetime import datetime as dt

console_out = "/STDOUT/Console.out"
chunksize = 50
lock = Lock()

def writer(message):
    lock.acquire()
    with open(console_out, 'a') as out:
        out.write(message)
        out.flush()
    lock.release()

def conf_wrapper(state):
    import ProcessingModule as procs
    import sqlalchemy as sal

    stcd, nrows = state
    engine = sal.create_engine('postgresql://foo:bar@localhost:5432/schema')

    writer("State {s} started  at: {n}"
           "\n".format(s=str(stcd).zfill(2), n=dt.now()))

    with engine.connect() as conn, conn.begin():
        procs.processor(conn, stcd, nrows, chunksize)

    writer("\tState {s} finished  at: {n}"
           "\n".format(s=str(stcd).zfill(2), n=dt.now()))

def main():
    nprocesses = 12
    maxproc = 1
    state_list = [(2, 113), (10, 119), (15, 84), (50, 112), (44, 110), (11, 37), (33, 197)]

    with open(console_out, 'w') as out:
        out.write("Starting at {n}\n".format(n=dt.now()))
        out.write("Using {p} processes..."
                  "\n".format(p=nprocesses))

    with Pool(processes=int(nprocesses), maxtasksperchild=maxproc) as pool:
        pool.map(func=conf_wrapper, iterable=state_list, chunksize=1)

    with open(console_out, 'a') as out:
        out.write("\nAll done at {n}".format(n=dt.now()))

The file console_out never has all 7 states in it. It always misses one or more state. Here is the output from the latest run:

Starting at 2016-07-27 21:46:58.638587
Using 12 processes...
State 44 started  at: 2016-07-27 21:47:01.482322
State 02 started  at: 2016-07-27 21:47:01.497947
State 11 started  at: 2016-07-27 21:47:01.529198
State 10 started  at: 2016-07-27 21:47:01.497947
    State 11 finished  at: 2016-07-27 21:47:15.701207
    State 15 finished  at: 2016-07-27 21:47:24.123164
    State 44 finished  at: 2016-07-27 21:47:32.029489
    State 50 finished  at: 2016-07-27 21:47:51.203107
    State 10 finished  at: 2016-07-27 21:47:53.046876
    State 33 finished  at: 2016-07-27 21:47:58.156301
    State 02 finished  at: 2016-07-27 21:48:18.856979

All done at 2016-07-27 21:48:18.992277

Why?

Note, OS is Windows Server 2012 R2.

Kartik
  • 8,347
  • 39
  • 73

1 Answers1

1

Since you're running on Windows, nothing is inherited by worker processes. Each process runs the entire main program "from scratch".

In particular, with the code as written every process has its own instance of lock, and these instances have nothing to do with each other. In short, lock isn't supplying any inter-process mutual exclusion at all.

To fix this, the Pool constructor can be changed to call a once-per-process initialization function, to which you pass an instance of Lock(). For example, like so:

def init(L):
    global lock
    lock = L

and then add these arguments to the Pool() constructor:

initializer=init, initargs=(Lock(),),

And you no longer need the:

lock = Lock()

line.

Then the inter-process mutual exclusion will work as intended.

WITHOUT A LOCK

If you'd like to delegate all output to a writer process, you could skip the lock and use a queue instead to feed that process [and see later for different version].

def writer_process(q):
    with open(console_out, 'w') as out:
        while True:
            message = q.get()
            if message is None:
                break
            out.write(message)
            out.flush() # can't guess whether you really want this

and change writer() to just:

def writer(message):
    q.put(message)

You would again need to use initializer= and initargs= in the Pool constructor so that all processes use the same queue.

Only one process should run writer_process(), and that can be started on its own as an instance of multiprocessing.Process.

Finally, to let writer_process() know it's time to quit, when it is time for it to drain the queue and return just run

q.put(None)

in the main process.

LATER

The OP settled on this version instead, because they needed to open the output file in other code simultaneously:

def writer_process(q):
    while True:
        message = q.get()
        if message == 'done':
            break
        else:
            with open(console_out, 'a') as out:
                out.write(message)

I don't know why the terminating sentinel was changed to "done". Any unique value works for this; None is traditional.

Tim Peters
  • 67,464
  • 13
  • 126
  • 132
  • What then happens is that `Pool` creates `lock` in the global namespace, which is shared by all worker processes, correct? – Kartik Jul 28 '16 at 05:25
  • 1
    _Nothing_ is shared across processes on Windows by magic. The `init()` function is _called_ in each process, and the latter binds the process-global name `lock` (because of the `global lock` statement) within each process to the single instance of `Lock()` created in - and passed by - the main process. – Tim Peters Jul 28 '16 at 05:28
  • Oh! I get it. If it is not asking too much, can you also write an answer that uses mp.Manager and Queue instead of primitives like Lock? That would be my preferred approach, but when I tried it, it didn't work at all. So I switched to using Lock, and inadvertently asked this question. – Kartik Jul 28 '16 at 05:36
  • If your current question has been answered, you should accept the answer, and open new issues for new questions. Which should say more about what you're trying to accomplish, because I have no idea what you'd use a `Manager` or `Queue` for in _this_ question: the only thing this question appeared to need was inter-process mutual exclusion, and that's exactly what an `mp.Lock()` is for. – Tim Peters Jul 28 '16 at 05:39
  • See edit: gave a semi-detailed sketch of how output could be handled by a queue instead. But still no idea what you'd want a `Manager` for. – Tim Peters Jul 28 '16 at 06:19
  • Thank you. I personally feel that using a queue is more elegant than setting locks. It's just the idea of using a separate process to handle the output and feeding that process seems more Pythonic. My main goal is to have a file logging the progress of the code. There was a time when I was simply printing the output on screen, but then I started using GDAL, which dumps a whole load of messages on the screen obfuscating the progress markers. Hence the writing to the file. (Ps. I was using `out.flush()` because I assumed the file was not being flushed, resulting in the missing lines.) – Kartik Jul 28 '16 at 08:30
  • And I also read that `Manager` helps with interprocess communication on Windows. Which I thought was lacking because my `Queue` approach did not work at all. – Kartik Jul 28 '16 at 08:33
  • 1
    I'd personally use a `mp.Queue` along the lines shown, but run it in a thread in the main process (no real need to create a new process for it); `t = threading.Thread(target=writer_process, args=(q,)); t.start()` and then when the program is ending `writer(None); t.join()`. `mp.Manager` facilities can be convenient, but using them incurs high interprocess communication overheads, so I've rarely found them "worth it" in the end. – Tim Peters Jul 28 '16 at 18:28
  • Awesome! Thank you for clarifying and helping with this. Sincerely appreciated. – Kartik Jul 28 '16 at 18:30
  • The `'done'` came because it was copied from elsewhere. I had assumed that `q.get()` on an empty queue might return `None`. – Kartik Jul 29 '16 at 19:12
  • 1
    No, `q.get()` when the queue is empty blocks until something is added to the queue. The blocking logic is blind to _what_ is added to the queue - it's only waiting for _something_ to be added. `None`, an empty string, a list with a billion elements ... doesn't matter what. – Tim Peters Jul 29 '16 at 19:18
  • So the `else` in my version is not required. Thank you for patiently teaching me all this. – Kartik Jul 29 '16 at 20:05
  • Right, I don't _think_ it's needed, but can't guess whether you sometimes pass `None` for some unknown (to me) reason. I'll edit again to take that part out. – Tim Peters Jul 29 '16 at 20:13
  • It is not needed. I tried it without the `else` and it works splendidly. I do not pass `None`, I just pass the two messages in my question: start and end times of each state. This is to monitor progress. Like I said, GDAL prints a whole load of crap to the console, and I needed another way to separate the progress statements from that. (Here is the associated question I asked: http://stackoverflow.com/questions/38601836/importing-gdal-prints-lots-of-error-messages-but-still-works) – Kartik Jul 29 '16 at 20:34
  • Thanks again Tim, you have no idea how useful this has been for me. If I could, I would upvote your answer 10 more times, at least. – Kartik Aug 11 '16 at 06:33