I am trying to use the pathos multiprocessing for a simple function for which I pass an array, it splits into parallel processes and then joins at the end.
Here is the code I ported over from using the multiprocessing function, and then instead of importing multiprocessing, I imported pathos.
import sys
import gender_guesser.detector as gender
import time
from pathos import multiprocessing as multiprocessing
d = gender.Detector()
def guess_gender (name):
n = name.title() # make first letter upper case and the rest lower case
g = d.get_gender(n) # guess gender
return g
ls = ['john','joe','amamda','derick','peter','ashley','john','joe','amamda','derick','peter','ashley']
t=time.time()
results=[]
def callBack(x):
results.append(x)
pool = multiprocessing.Pool(processes=multiprocessing.cpu_count()-1, maxtasksperchild=1)
for n in ls:
print (n)
pool.apply_async(guess_gender,args=[n],callback=callBack)
pool.close()
pool.join()
results = pd.concat(results)
print(time.time()-t)
However, I get this error:
ValueError: No objects to concatenate
I am not sure how I extract the results of the output. The documentation on pathos is thin.
Does anyone have any ideas or experience?
FYI i am running python in jupyter notebooks.
