I've looked at Can't pickle <type 'instancemethod'> when using python's multiprocessing Pool.map() which describes a similar problem, but I'm not using a class' method.
What I'm trying to do is update several MySQL servers concurrently. I have:
def do_update(cur, update, output_fh):
try:
cur.execute(update)
except MySQLdb.MySQLError, e:
output_fh.write("failed on:{}\n".format(update))
raise MySQLdb.MySQLError(e)
to perform the actual update. I've attempted two different approaches to use a multiprocessing Pool to update in parallel and both result in the error message in the subject.
1:
def update_score_wrapper(args):
do_update(*args)
pool = multiprocessing.Pool(processes=3)
pool.map(update_score_wrapper, ((cur2, update, output_fh),
(cur3, update, output_fh), (cur4, update, output_fh)))
where cur2, cur3, and cur4 are MySQLdb cursors, update is the update statement, and output_fh is a file handle to a shared log.
2:
partial_score = functools.partial(
do_update, update=update, output_fh=output_fh)
pool = multiprocessing.Pool(processes=3)
pool.map(partial_score, (cur2, cur3, cur4))
Anyone able to tell me what's going wrong here? My suspicion is that perhaps the cursor or file handle is not picklable? If so, are there any alternative approaches to try?
