joblib parallel processing of a multiple return values function

Question

I use joblib to parallelise a function (with multiprocessing). But, this function return 4 values but when I get the results from Parallel it gives me only 3 values

from joblib import Parallel, delayed 
import numpy as np
from array import array
import time

def best_power_strategy():
    powerLoc = {0}
    speedLoc = {1}
    timeLoc = {2}
    previousSpeedLoc = {3}        
    return powerLoc,speedLoc,timeLoc,previousSpeedLoc

if __name__ == "__main__":
    realRiderName=['Rider 1', 'Rider 2', 'Rider 3']
    powerLoc = {}
    speedLoc = {}
    timeLoc = {}
    previousSpeedLoc = {}
    powerLoc,speedLoc,timeLoc,previousSpeedLoc = Parallel(n_jobs=3)(delayed(best_power_strategy)() for rider in realRiderName)
    print(powerLoc)
    print(speedLoc)
    print(timeLoc)
    print(previousSpeedLoc)

and the result is :

ValueError: not enough values to unpack (expected 4, got 3)

Does someone has an idea?

Thanks in advance

You're returning a generator calling `best_power_strategy()` for each `realRiderName`. Since there are three elements in `realRiderName`, that's how many results are you getting. — zwer, Jul 30 '18 at 16:52
Ah ok, so how can I return powerLoc,speedLoc,timeLoc and previousSpeedLoc in parallel? — Doctor Pi, Jul 30 '18 at 17:00
So if I do `res = Parallel(n_jobs=3)(delayed(best_power_strategy)() for rider in realRiderName)` `print(res)` I got ` [({0}, {1}, {2}, {3}), ({0}, {1}, {2}, {3}), ({0}, {1}, {2}, {3})]` And then how i split this list into 4 lists (one for each number)? — Doctor Pi, Jul 30 '18 at 17:17

score 12 · Answer 1 · answered Jul 30 '18 at 18:46

If you want to store the results in four separate names, you can zip the results from your generator together and then expand them into the desired names, i.e.:

# shortening the names for simplicity/readability
riders = ["Rider 1", "Rider 2", "Rider 3"]
p, s, t, pv = zip(*Parallel(n_jobs=3)(delayed(best_power_strategy)() for r in riders))

This will result in p containing all the powerLoc results, s containing all the speedLoc results and so on...

Now, given that your best_power_strategy function is essentially static and nothing changes (you're not even sending a rider to it), this piece of code is pretty useless as you'll always have the same results, but I take it you're using this just as an example.

This code works but zip is realy slow on the real code – Doctor Pi Jul 30 '18 at 19:20 — Doctor Pi, Jul 30 '18 at 19:20

score 4 · Answer 2 · answered Jul 30 '18 at 18:27

Ok I've solved the problem:

from joblib import Parallel, delayed 
import numpy as np
from array import array
import time

def best_power_strategy():
    powerLoc = {0}
    speedLoc = {1}
    timeLoc = {2}
    previousSpeedLoc = {3}

    return powerLoc,speedLoc,timeLoc,previousSpeedLoc

if __name__ == "__main__":
    realRiderName=['Rider 1', 'Rider 2', 'Rider 3']
    powerLoc = {}
    speedLoc = {}
    timeLoc = {}
    previousSpeedLoc = {}
    res = Parallel(n_jobs=3)(delayed(best_power_strategy)() for rider in realRiderName)
    powerLoc=[item[0] for item in res]
    speedLoc=[item[1] for item in res]
    timeLoc=[item[2] for item in res]
    previousSpeedLoc=[item[3] for item in res]

    print(powerLoc)
    print(speedLoc)
    print(timeLoc)
    print(previousSpeedLoc)

Why are you calling `best_power_strategy` three times if the result of each call is exactly the same? — TayTay, Jul 30 '18 at 18:33
It is just a MWE, of course the real code is a lot more complex — Doctor Pi, Jul 30 '18 at 19:18

joblib parallel processing of a multiple return values function

2 Answers2