Parallelize this code

Question

I am trying to figure out how to parallelize the following code. I have looked up joblib, concurrent.futures, and multiprocessing modules, but cannot for the life of me figure out how they work from reading the docs and scouring SO/google.

Grid is a class object with the appropriate methods defined and it does not matter what order the loop is processed.

def ProcessGrid(Grid):
    #Parallel This Loop
    for i in range(len(Grid)):
        Grid[i].AdjustCostMultiplier()
        Grid[i].FindAllNeighbours(Grid)
        print("Processing Grid Cell: " + str(i) + " of " + str(len(Grid)),end = "\r")
    #Return to serial
    return Grid

Do you want to call the ProcessGrid method 2 times parallel? Or do you want to call AdjustCostMultiplier() and FindAllNeighbours()? — Tim Woocker, Mar 29 '18 at 21:05
I want to parallelize the for loop, I'll edit my question to reflect the clarification — Mark Kavanagh, Mar 29 '18 at 21:09
"Do X for me" requests (it's hard to describe them as "questions") don't tend to get a good response here without at least showing your own effort -- not just asserting that you tried, but actually showing the code you wrote in the form of a [mcve] and describing how it fails. — Charles Duffy, Mar 29 '18 at 21:13
Alright, I added an answer. I didn't test it tho but it should work :) — Tim Woocker, Mar 29 '18 at 21:18

Tim Woocker · Accepted Answer · 2018-03-29T21:36:30.123

2

You can use the threading library like this:

import threading

def process(grid, i):
    grid[i].AdjustCostMultiplier()
    grid[i].FindAllNeighbours(Grid)
    print("Processing Grid Cell: " + str(i) + " of " + str(len(grid)), end = "\r")

def ProcessGrid(Grid):
    threads = []
    for i in range(len(Grid)):
        t = threading.Thread(target=process, args=(Grid, i))
        t.start()
        threads.append(t)
    for t in threads:
        # Wait for all threads to finish
        t.join()
    #Return to serial
    return Grid

process() will be called in a new thread for every iteration. t.join() then waits for the threads to finish.

edited Mar 29 '18 at 21:36

answered Mar 29 '18 at 21:17

Tim Woocker

1,883
1
16
29

Python multi-threading does not make code faster when it is CPU bound, unfortunately, because of the Global Interpreter Lock. Multiprocessing does make it faster, but it will take a bit more effort. – Maarten-vd-Sande Mar 29 '18 at 21:28
He asked to parallelize it tho. He never asked to make his code faster. Also multithreading can make code faster if there is a waiting process involved. – Tim Woocker Mar 29 '18 at 21:32
there was some issues with t being "NoneType' (I guess because of no return statements?), but I just commented out the join statement and it worked fine for my case – Mark Kavanagh Mar 29 '18 at 21:35
oops, sorry I fixed it. I stored .start() into t instead of the thread itself. Just move t.start() into the line below. – Tim Woocker Mar 29 '18 at 21:37
1

@Maarten_vd_Sande Parallelizing can make code faster if (a) it's not CPU-bound, or (b) it is CPU-bpund. but most of that work is inside C extension libraries that release the GIL for major operations, like numpy, or (c) it's run in a GIL-less Python interpreter like Jython. – abarnert Mar 30 '18 at 00:56

Parallelize this code

1 Answers1