4

I have a list (called requestRoster) containing dictionaries (called requests). Items in the 'requests' dictionary are things like 'requestTime' and 'thisURL'. E.g.: 

[
{'thisURL': 'http://localhost/bikes', 'requestTime': datetime.datetime(2012, 10, 18, 0, 41, 34)}, 
{'thisURL': 'http://localhost/clothing', 'requestTime': datetime.datetime(2012, 10, 18, 0, 41, 35)}
]

I am using multiprocessing.Process to spawn a new process to issue each request.

I would like each process to update the requestRoster, adding a 'response' item to each request.

How can I do this?

I have tried using a multiprocessing.Manager() to make a manager.list() and a manager.Namespace(). Neither lets me do what I want to do, I think because of this: http://docs.python.org/library/multiprocessing.html#multiprocessing.managers.SyncManager.list

I think I could use a multiprocessing.Lock() to

  • acquire a mutex
  • make a copy of the requestRoster inside the process
  • modify the localised requestRoster
  • overwrite the 'globablised' request roster with the localised one
  • release the mutex

... but it seems a bit elaborate and I wonder if I'm missing something simpler. An asynchronous callback would be great.

dave
  • 572
  • 7
  • 16
  • Do your workers need to read from the roster, or just write responses to it? From your description, it seems like a simple queue or even a pipe hooked up between the dispatcher and each worker should work to push your response data back, and then the dispatcher can be responsible for logging the response to the right entry in your call history. – Silas Ray Oct 23 '12 at 15:14
  • Okay, a queue. So the dispatcher dispatches; the worker works; the worker adds a result to a queue; the dispatcher considers the queue and dequeues responses. Something like that? I like that... the only thing I'm wary of is making the dispatcher responsible for anything more than dispatching as it is already overwhelmed with work when some hundreds of requests have to be dispatched simultaneously. I think there is a genuine case for the worker process to have access to a 'global' data structure... and for the workers to do all the work, including updating. – dave Oct 23 '12 at 15:51
  • This is growing on me. I could get the dispatcher to dequeue only when it has time - but focus on dispatching most of the time. Thanks for your input. – dave Oct 23 '12 at 15:54
  • You can always push the logging off on to a logging manager thread/process and have the workers push response data to the logger. The real point is the compartmentalization of the collating/logging task and the request/response tasks. – Silas Ray Oct 23 '12 at 16:01
  • Wouldn't the logging process also suffer the same interprocess-writing problems as the HTTP worker processes? I.e. it would be unable to update the main requestRoster. But I could drain the queue after everything has been dispatched. Compartmentalise indeed. – dave Oct 23 '12 at 16:13
  • Nope, having thought about this more, I'd still like to globalise the structure, I really want to keep the dispatcher lean and I really want results to be committed in real time. I'm using Pyside, maybe QThread could help? – dave Oct 23 '12 at 16:15
  • What ultimately needs to be done with the roster? The dispatcher can have a pipe or queue open to the logger, create a pipe for each worker, push the pipe to the worker to the logger which can then attach to it, and then the workers can push their responses to the pipe the dispatcher set up. This should work, I think, unless you need some other workers/processes reading from the log at the same time. – Silas Ray Oct 23 '12 at 16:16
  • The roster will be displayed in a Qt grid (of some sort) with results appearing in real-time. – dave Oct 23 '12 at 16:23

3 Answers3

2

It's nicer to avoid shared memory structures, if you can. Here, there's no reason for you to have the processes write to the list of dicts themselves -- instead, you could make the main process responsible to this and farm out only the URL fetch to the processes.

I like concurrent.futures.<Process|Thread>PoolExecutor for this sort of thing.

Katriel
  • 120,462
  • 19
  • 136
  • 170
  • Thanks for your input, but I'm on Python 2.7.3 so I don't think I can use concurrent.futures yet (although it looks good). I think it would be better if the child processes (or whatever you want to call them) writes to the list of dicts because the main process is already overwhelmed with work - for instance, when some hundreds of requests have to be dispatched 'simultaneously'. – dave Oct 23 '12 at 15:46
0

I think this approach should work for you:

Dispatcher:

create logger_queue
create logger process, initialize with logger_queue
for each request
    create worker_pipe
    create worker process, initialize with send end of worker_pipe
    push receive end of worker_pipe over logger_queue

Worker:

make request
push response over connection

Logger:

while True
    for connection on logger_queue
        create new element in logging list
        link connection to new logging list element
    for each open connection
        poll for message
        if message
            store message to log
            close connection

The logger process can also be running whatever output routines you want, so you don't even have to worry about having another process reading from the logged dataset. Note that connection above refers to multiprocessing.Connection.

Silas Ray
  • 25,682
  • 5
  • 48
  • 63
  • Input very much appreciated, but massively complex considering I just want to globalise a data structure. I don't really want to make queues, pipes and loggers (more problems) I just want the workers to be able to update data in the main process... (but thanks again for your input, it is appreciated) – dave Oct 23 '12 at 16:49
  • Shared datastructures are very complex, and have lots of overhead and pitfalls. If you can avoid them, you probably should. If you just want to do it all in the main process, put the logger logic in the dispatcher and cut out the queue. I broke it out this way because you said you wanted the dispatcher lean. – Silas Ray Oct 23 '12 at 16:55
0

I managed to do this by using threading instead of multiprocessing. Because the workers are within the same process as the dispatcher they can update the requestRoster.

dave
  • 572
  • 7
  • 16
  • Threading will probably be fine for you since your workers are I/O-bound, but be aware that should you ever run in to CPU-bound performance problems, the GIL will force you to move to multiprocess instead. – Silas Ray Oct 24 '12 at 16:03