recover values from ipyparallel map if a single evaluation failed

Question

I run a quite time consuming (>3 days) simulation using ipyparallel's map function like that in Jupyter Notebook

from ipyparallel import Client
rc = Client()
lview = rc.load_balanced_view()
ar = lview.map(runsimulation, parameter)

and I really need those results. But unfortunately a single instance died because of memory starvation. Now i can not access the result using ar[i] anymore. Is there a chance to recover all the other results that where most certainly computed (I can see the characteristic memory and cpu time consumption in the Ganglia Monitor of the cluster).

I still have the object ar in the Jupyter Notebook.

score 2 · Answer 1 · answered Oct 18 '16 at 11:21

You can get partial results for individual msg_ids. With map, each message corresponds to a chunk of the map (a list):

import ipyparallel as ipp

results = []
for msg_id in ar.msg_ids:
    chunk_ar = rc.get_result(msg_id)
    try:
        chunk = chunk_ar.get()
    except ipp.RemoteError as e:
        print('ignoring error: %s' % e)
    results.extend(chunk)

score 0 · Answer 2 · answered Oct 20 '16 at 15:23

For me the answer given by minrk didn't work, because it failed already on get_result.

I modified it a bit and this worked for me:

results = []
for msg_id in ar.msg_ids:
    try:
        chunk_ar = rc.get_result(msg_id)
        chunk = chunk_ar.get()
    except:
        chunk = None
    results.append(chunk)

Note that I add None, such that I get an equal length list as the inputs.

recover values from ipyparallel map if a single evaluation failed

2 Answers2