I have many independent tasks that read but not write to the same gensim model which is about 3.6GB in size. (Gensim is a topic modelling library built upon numpy.) So I decide to parallelize them by first loading the gensim model from a file:
from gensim.models.word2vec import Word2Vec
from multiprocessing import Pool
model = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
And then pass the model as a parameter to a pool of processes to run doWork:
def doWork(experiment, doc):
print "Begin working"
# do some work here; access model by experiment.model
class Experiment(object):
def __init__(self, model, docs):
self.model = model
self.docs = docs
def run(self):
pool = Pool(processes = 4)
print "Done preparing"
results = pool.map(doWork, [(self, doc) for doc in self.docs])
return results
experiment = Experiment(model, ['doc1.txt', 'doc2.txt'])
experiment.run()
When I ran this script (the two segments I show here are a runnable script; please copy), it got stuck on the pool.map line and a SystemError occurred. The output was:
Done preparing
Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
SystemError: NULL result without error in PyObject_Call
The error had never occurred before I introduced gensim to my program. (Gensim without multiprocessing also works for me.) I think it may be related to the interoperation with C code underlying gensim and numpy (BLAS). I wanted to know the reason of this error and how to fix it. If I can't use gensim with subprocessing, what are the alternatives?
I don't think the model would be copied, because my OS (Mac OS X) should be using the copy-on-write strategy. I don't think it is related to memory synchronization either, because not a line of "Begin working" is printed, i.e. the model has not been accessed by my code. The error is in passing the model to the sub-processes.
