Python multiprocessing design

Question

I have written an algorithm that takes geospatial data and performs a number of steps. The input data are a shapefile of polygons and covariate rasters for a large raster study area (~150 million pixels). The steps are as follows:

1. Sample points from within polygons of the shapefile
2. For each sampling point, extract values from the covariate rasters
3. Build a predictive model on the sampling points
4. Extract covariates for target grid points
5. Apply predictive model to target grid
6. Write predictions to a set of output grids

The whole process needs to be iterated a number of times (say 100) but each iteration currently takes more than an hour when processed in series. For each iteration, the most time-consuming parts are step 4 and 5. Because the target grid is so large, I've been processing it a block (say 1000 rows) at a time.

I have a 6-core CPU with 32 Gb RAM, so within each iteration, I had a go at using Python's multiprocessing module with a Pool object to process a number of blocks simultaneously (steps 4 and 5) and then write the output (the predictions) to the common set of output grids using a callback function that calls a global output-writing function. This seems to work, but is no faster (actually, it's probably slower) than processing each block in series.

So my question is, is there a more efficient way to do it? I'm interested in the multiprocessing module's Queue class, but I'm not really sure how it works. For example, I'm wondering if it's more efficient to have a queue that carries out steps 4 and 5 then passes the results to another queue that carries out step 6. Or is this even what Queue is for?

Any pointers would be appreciated.

Answer 1

The current state of Python's multi-processing capabilities are not great for CPU bound processing. I fear to tell you that there is no way to make it run faster using the multiprocessing module nor is it your use of multiprocessing that is the problem.

The real problem is that Python is still bound by the rules of the GlobalInterpreterLock(GIL) (I highly suggest the slides). There have been some exciting theoretical and experimental advances on working around the GIL. Python 3.2 event contains a new GIL which solves some of the issues, but introduces others.

For now, it is faster to execute many Python process with a single serial thread than to attempt to run many threads within one process. This will allow you avoid issues of acquiring the GIL between threads (by effectively having more GILs). This however is only beneficial if the IPC overhead between your Python processes doesn't eclipse the benefits of the processing.

Eli Bendersky wrote a decent overview article on his experiences with attempting to make a CPU bound process run faster with multiprocessing.

It is worth noting that PEP 371 had the desire to 'side-step' the GIL with the introduction of the multiprocessing module (previously a non-standard packaged named pyProcessing). However the GIL still seems to play too large of a role in the Python interpreter to make it work well with CPU bound algorithms. Many different people have worked on removing/rewriting the GIL, but nothing has made enough traction to make it into a Python release.

Answer 2

Some of the multiprocessing examples at python.org are not very clear IMO, and it's easy to start off with a flawed design. Here's a simplistic example I made to get me started on a project:

import os, time, random, multiprocessing
def busyfunc(runseconds):
    starttime = int(time.clock())
    while 1:
        for randcount in range(0,100):
            testnum = random.randint(1, 10000000)
            newnum = testnum / 3.256
        newtime = int(time.clock())
        if newtime - starttime > runseconds:
            return

def main(arg):
    print 'arg from init:', arg
    print "I am " + multiprocessing.current_process().name

    busyfunc(15)

if __name__ == '__main__':

    p = multiprocessing.Process(name = "One", target=main, args=('passed_arg1',))
    p.start()

    p = multiprocessing.Process(name = "Two", target=main, args=('passed_arg2',))
    p.start()

    p = multiprocessing.Process(name = "Three", target=main, args=('passed_arg3',))
    p.start()

    time.sleep(5)

This should exercise 3 processors for 15 seconds. It should be easy to modify it for more. Maybe this will help to debug your current code and ensure you are really generating multiple independent processes.

If you must share data due to RAM limitations, then I suggest this: http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes

Answer 3

As python is not really meant to do intensive number-cunching, I typically start converting time-critical parts of a python program to C/C++ and speed things up a lot.

Also, the python multithreading is not very good. Python keeps using a global semaphore for all kinds of things. So even when you use the Threads that python offers, things won't get faster. The threads are useful for applications, where threads will typically wait for things like IO.

When making a C module, you can manually release the global semaphore when processing your data (then, of course, do not access the python values anymore).

It takes some practise using the C API, but's its clearly structured and much easier to use than, for example, the Java native API.

See 'extending and embedding' in the python documentation.

This way you can make the time critical parts in C/C++, and the slower parts with faster programming work in python...

Answer 4

I recommend you first check which aspects of your code is taking the most time, so your gonna have to profile it, I've used http://packages.python.org/line_profiler/#line-profiler with much success, though it does require cython.

As for Queues, their mostly used for sharing data/synchronizing threads, though I've rarely used it. I do use multiprocessing all the time.

I mostly follow the map reduce philosophy, which is simple and clean but it has some major overhead, since values have to be packed into dictionaries and copied across each process, when applying the map function ...

You can try segmenting your file and applying your algorithm to different sets.