0

I'm very new to Python (and coding in general) and I need help parallising the code below. I looked around and found some packages (eg. Multiprocessing & JobLib) which could be useful.

However, I have trouble using it in my example. My code makes an outputfile, and updates it doing the loop(s). Therefore is it not directly paralisable, so I think I need to make smaller files. After this, I could merge the files together.

I'm unable to find a way to do this, could someone be so kind and give me a decent start?

I appreciate any help, A code newbie

Code:

def delta(graph,n,t,nx,OutExt):
    fout_=open(OutExt+'Delta'+str(t)+'.txt','w')
    temp=nx.Graph(graph)
    for u in range(0,n):
        #print "stamp: "+str(t)+" node: "+str(u)
        for v in range(u+1,n):
            #print str(u)+"\t"+str(v)
            Stat = dict()
            temp.add_edge(u,v)
            MineDeltaGraphletTransitionsFromDynamicNetwork(graph,temp,Stat,u,v)
            for a in Stat:
                for b in Stat[a]:
                    fout_.write(str(t)+"\t"+str(u)+"\t"+str(v)+"\t"+str(a)+"\t"+str(b)+"\t"+str(Stat[a][b])+"\n")
            if not graph.has_edge(u,v):
                temp.remove_edge(u,v)
    del temp
    fout_.close()
thepunitsingh
  • 713
  • 1
  • 12
  • 30
Tim D
  • 111
  • 1
  • 2
  • 7
  • Note that if your data is small enough to fit in memory, you don't have to write to the file immediately. Each thread can keep its own list of results, and you can write these results to the file once all threads have joined. – 0x5453 Mar 12 '18 at 13:24
  • I was thinking this way too, but I dont know how I could do this. Could you provide me an example please? – Tim D Mar 12 '18 at 13:30

2 Answers2

2

As a start, find the part of the code that you want to be able to execute in parallel with something (perhaps with other invocations of that very same function). Then, figure out how to make this code not share mutable state with anything else.

Mutable state is the enemy of parallel execution. If two pieces of code are executing in parallel and share mutable state, you don't know what the outcome will be (and the outcome will be different each time you run the program). This is becaues you don't know what order the code from the parallel executions will run in. Perhaps the first will mutate something and then the second one will compute something. Or perhaps the second one will compute something and then the first one will mutate it. Who knows? There are solutions to that problem but they involve fine-grained locking and careful reasoning about what can change and when.

After you have an algorithm with a core that doesn't share mutable state, factor it into a separate function (turning locals into parameters).

Finally, use something like the threading (if your computations are primarily in CPython extension modules with good GIL behavior) or multiprocessing (otherwise) modules to execute the algorithm core function (which you have abstracted out) at some level of parallelism.

The particular code example you've shared is a challenge because you use the NetworkX library and a lot of shared mutable state. Each iteration of your loop depends on the results of the previous, apparently. This is not obviously something you can parallelize. However, perhaps if you think about your goals more abstractly you will be able to think of a way to do it (remember, the key is to be able to expressive your algorithm without using shared mutable state).

Your function is called delta. Perhaps you can split your graph into sub-graphs and compute the deltas of each (which are now no longer shared) in parallel.

If the code within your outermost loop is concurrent safe (I don't know if it is or not), you could rewrite it like this for parallel execution:

from multiprocessing import Pool

def do_one_step(nx, graph, n, t, OutExt, u):
    # Create a separate output file for this set of results.
    name = "{}Delta{}-{}.txt".format(OutExt, t, u)
    fout_ = open(name, 'w')
    temp = nx.Graph(graph)

    for v in range(u+1,n):
        Stat = dict()
        temp.add_edge(u,v)
        MineDeltaGraphletTransitionsFromDynamicNetwork(graph,temp,Stat,u,v)
        for a in Stat:
            for b in Stat[a]:
                fout_.write(str(t)+"\t"+str(u)+"\t"+str(v)+"\t"+str(a)+"\t"+str(b)+"\t"+str(Stat[a][b])+"\n")
        if not graph.has_edge(u,v):
            temp.remove_edge(u,v)
    fout_.close()

def delta(graph,n,t,nx,OutExt):
    pool = Pool()
    pool.map(
        partial(
            do_one_step,
            nx,
            graph,
            n,
            t,
            OutExt,
        ),
        range(0,n),
    )

This supposes that all of the arguments can be serialized across processes (required for any argument you pass to a function you call with multiprocessing). I suspect that nx and graph may be problems but I don't know what they are.

And again, this assumes it's actually correct to concurrently execute the inner loop.

Jean-Paul Calderone
  • 47,755
  • 6
  • 94
  • 122
  • Hi Jean-Paul and thank you for your crisp explaination. Follow up question, do you think everything in the loop `for u in range(0,n):` could be parallizable in one step? Could it cause problems with the other for loops within? I know I need to edit the line `fout_.write` for this. – Tim D Mar 12 '18 at 14:00
  • The code inside that loop calls `temp.add_edge` and `temp.remove_edge` (also `MineDeltaGraphletTransitionsFromDynamicNetwork` which is not included so I don't know what it does). Since `temp` is mutated (perhaps a lot) inside the `for u in range(0,n):` loops it doesn't seem obviously parallelizable. What happens if the iteration where `u=1` runs before the iteration where `u=0`? That can happen if you parallelize at this level without further refactoring. If you _know_ that is safe for your algorithm, great, you can do it. It doesn't look safe to me but I don't know the goal of this function. – Jean-Paul Calderone Mar 12 '18 at 14:13
  • Actually, `for u in range(0,n):` -loop could perfectly run indepedant. What is important that the inner loops not are mixed up (this wont happen I presume?). – Tim D Mar 12 '18 at 18:02
  • What do you mean "mixed up"? – Jean-Paul Calderone Mar 12 '18 at 18:03
  • Sorry for being unclear. What I mean is that within each subprocess of the loop, the order of the other loops and functions is respected. So only the `for u in range(0,n)` is parallised, and not again the loops within. – Tim D Mar 12 '18 at 18:17
  • For the most part, the rules about how code executes in the subprocess are the same as the normal rules. The subprocess is just another Python process running some Python code. So, yes, execution starts at the top and proceeds downwards, with function calls, loops, and other control structures retaining their normal meaning. – Jean-Paul Calderone Mar 12 '18 at 18:22
-2

Best use pool.map. Here an example that shows what you need to do. Here a simple example of how multiprocessing works with pool:

Single threaded, basic function:

def f(x):
    return x*x

if __name__ == '__main__':
     print(map(f, [1, 2, 3]))

>> [1, 4, 9]

Using multiple processors:

from multiprocessing import Pool 

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(3) # 3 parallel pools
    print(p.map(f, [1, 2, 3]))

Using 1 processor

from multiprocessing.pool import ThreadPool as Pool 

def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(3) # 3 parallel pools
    print(p.map(f, [1, 2, 3]))

When you use map you can easily get a list back from the results of your function.

Nickpick
  • 6,163
  • 16
  • 65
  • 116
  • Two questions: - How could I use different core's? (this is what I'm looking for) - How does the variable "threads" work? Could you provide me the link where you found the example? Tx! – Tim D Mar 12 '18 at 13:29
  • @Jean-PaulCalderone corrected that. No more global. – Nickpick Mar 12 '18 at 15:21