0

Looking to print everything in order, for a Python parallelized script. Note the c3 is printed prior to the b2 -- out of order. Any way to make the below function with a wait feature? If you rerun, sometimes the print order is correct for shorter batches. However, looking for a reproducible solution to this issue.

from joblib import Parallel, delayed, parallel_backend
import multiprocessing

testFrame = [['a',1], ['b', 2], ['c', 3]]

def testPrint(letr, numbr):
  print(letr + str(numbr))
  return letr + str(numbr)

with parallel_backend('multiprocessing'):
  num_cores = multiprocessing.cpu_count()
  results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0], 
        numbr = testFrame[i][1]) for i in range(len(testFrame))) 

print('##########')
for test in results:
  print(test)

Output:

b2
c3
a1
##########
a1
b2
c3

Seeking:
a1
b2
c3
##########
a1
b2
c3
Bob Hopez
  • 773
  • 4
  • 10
  • 28
  • 2
    Either in order or parallel. Your choice. – Klaus D. Mar 11 '19 at 19:45
  • Turn the list items into tuples with their indices (`['a',1] -> (0,['a',1])`) and pass that as the argument then return the index with the value and use it to reorder. Refactor the function to accept the new argument. – wwii Mar 11 '19 at 19:53
  • @wwii Want to try to crack it? – Bob Hopez Mar 11 '19 at 20:26
  • @KlausD. #What? – Bob Hopez Mar 11 '19 at 20:27
  • That's the implication: if you run things in parallel they will loose their order. – Klaus D. Mar 11 '19 at 21:14
  • @KlausD. In R, `doParallel` package, there's an ordered option. What do you think of @wwii. comment above? – Bob Hopez Mar 11 '19 at 21:26
  • 3
    His comment does not oppose mine. He just added information to reconstruct the original order in the result. It will have no effect on the prints. – Klaus D. Mar 11 '19 at 21:30
  • @KlausD. Thanks. Maybe the easiest is to set the print statements as elements in a nested list. ?? – Bob Hopez Mar 11 '19 at 21:37
  • 1
    I don't have `joblib` installed and my functions won't print in the main process if the are running in a separate process (using `multiprocessing` or `concurrent.futures.ProcessPoolExecutor`). If you are looking for ordered *execution*, don't parallelize it - like @KlausD. said. – wwii Mar 11 '19 at 23:13

1 Answers1

1

Once you launch tasks in separate processes you no longer control the order of execution so you cannot expect the actions of those tasks to execute in any predictable order - especially if the tasks can take varying lengths of time.

If you are parallelizing(?) a task/function with a sequence of arguments and you want to reorder the results to match the order of the original sequence you can pass sequence information to the task/function that will be returned by the task and can be used to reconstruct the original order.

If the original function looks like this:

def f(arg):
    l,n = arg
    #do stuff
    time.sleep(random.uniform(.1,10.))
    result = f'{l}{n}'
    return result

Refactor the function to accept the sequence information and pass it through with the return value.

def f(arg):
    indx, (l,n) = arg
    time.sleep(random.uniform(.1,10.))
    result = (indx,f'{l}{n}')
    return result

enumerate could be used to add the sequence information to the sequence of data:

originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)

Now the arguments have the form (index,originalarg) ... (0, ('a',0'), (1, ('b',1)).

And the returned values from the multi-processes look like this (if collected in a list) -

[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]

Which is easily sorted on the first item of each result, key=lambda item: item[0], and the values you really want obtained by picking out the second items after sorting results = [item[1] for item in results].

wwii
  • 23,232
  • 7
  • 37
  • 77
  • Thanks for the excellent description. If you can answer in the context of the code I shared, will mark as correct. – Bob Hopez Mar 12 '19 at 17:53
  • I cannot: I do not have `joblib` installed and don't have any way of testing a solution. No problem, don't accept if it doesn't answer your question - I wasn't sure that is what you were after anyway. – wwii Mar 12 '19 at 18:34
  • No need to install locally, works out of the box in https://colab.research.google.com – Bob Hopez Mar 13 '19 at 17:05