4

I need to convert a threading application to a multiprocessing application for multiple reasons (GIL, memory leaks). Fortunately the threads are quite isolated and only communicate via Queue.Queues. This primitive is also available in multiprocessing so everything looks fine. Now before I enter this minefield I'd like to get some advice on the upcoming problems:

  1. How to ensure that my objects can be transfered via the Queue? Do I need to provide some __setstate__?
  2. Can I rely on put returning instantly (like with threading Queues)?
  3. General hints/tips?
  4. Anything worthwhile to read apart from the Python documentation?
Helmut Grohne
  • 6,578
  • 2
  • 31
  • 67
  • Initializing database connections in `subclass(Process).__init__` *is* a problem. :-/ – Helmut Grohne Nov 04 '10 at 14:48
  • 1
    another user recently rewrote my threaded example with the multiprocessing library. i was using queue as well. might be worth a look: http://stackoverflow.com/questions/4072816/ideal-thread-structure-question-involves-multiple-thread-communication – mix Nov 09 '10 at 07:37

1 Answers1

5

Answer to part 1:

Everything that has to pass through a multiprocessing.Queue (or Pipe or whatever) has to be picklable. This includes basic types such as tuples, lists and dicts. Also classes are supported if they are top-level and not too complicated (check the details). Trying to pass lambdas around will fail however.

Answer to part 2:

A put consists of two parts: It takes a semaphore to modify the queue and it optionally starts a feeder thread. So if no other Process tries to put to the same Queue at the same time (for instance because there is only one Process writing to it), it should be fast. For me it turned out to be fast enough for all practical purposes.

Partial answer to part 3:

  • The plain multiprocessing.queue.Queue lacks a task_done method, so it cannot be used as a drop-in replacement directly. (A subclass provides the method.)
  • The old processing.queue.Queue lacks a qsize method and the newer multiprocessing version is inaccurate (just keep this in mind).
  • Since filedescriptors normally inherited on fork, care needs to be taken about closing them in the right processes.
Helmut Grohne
  • 6,578
  • 2
  • 31
  • 67