How do I get ipyparallel to print status updates while in wait_interactive() in Jupyterlab?

Question

I have the following code blocks, each in JupyterLab (this works as expected in Jupyter, but not Jupyter-Lab):

def work(sleepTime):
    import time
    import datetime
    start = datetime.datetime.now()
    time.sleep(sleepTime)
    return "Started at: "+str(start)+" Slept "+str(sleepTime)+" Ended at: "+str(datetime.datetime.now())

import ipyparallel as ipp
with ipp.Cluster(n=3) as rc:
    loadBalancedView = rc.load_balanced_view()
    ar = loadBalancedView.map(work,[10, 20, 30, 10, 20, 30, 10, 20, 30, 10, 20, 30, 10, 20, 30, 40, 1, 2, 3, 4, 10, 10, 10])
    ar.wait_interactive()
    ipp_result = ar.get()

ipp_result

Everything works as expected - the output from the ipp_result cell is:

['Started at: 2022-05-31 17:51:12.078368 Slept 10 Ended at: 2022-05-31 17:51:22.088425',
 'Started at: 2022-05-31 17:51:12.078694 Slept 20 Ended at: 2022-05-31 17:51:32.079246',
 'Started at: 2022-05-31 17:51:12.082793 Slept 30 Ended at: 2022-05-31 17:51:42.100084',
 'Started at: 2022-05-31 17:51:22.203872 Slept 10 Ended at: 2022-05-31 17:51:32.213932',
 'Started at: 2022-05-31 17:51:32.193991 Slept 20 Ended at: 2022-05-31 17:51:52.214057',
 'Started at: 2022-05-31 17:51:32.217268 Slept 30 Ended at: 2022-05-31 17:52:02.247332',
 'Started at: 2022-05-31 17:51:42.214434 Slept 10 Ended at: 2022-05-31 17:51:52.223233',
 'Started at: 2022-05-31 17:51:52.217649 Slept 20 Ended at: 2022-05-31 17:52:12.235251',
 'Started at: 2022-05-31 17:51:52.226401 Slept 30 Ended at: 2022-05-31 17:52:22.256461',
 'Started at: 2022-05-31 17:52:02.250662 Slept 10 Ended at: 2022-05-31 17:52:12.260710',
 'Started at: 2022-05-31 17:52:12.238504 Slept 20 Ended at: 2022-05-31 17:52:32.258577',
 'Started at: 2022-05-31 17:52:12.263679 Slept 30 Ended at: 2022-05-31 17:52:42.293737',
 'Started at: 2022-05-31 17:52:22.260061 Slept 10 Ended at: 2022-05-31 17:52:32.270130',
 'Started at: 2022-05-31 17:52:32.261914 Slept 20 Ended at: 2022-05-31 17:52:52.281975',
 'Started at: 2022-05-31 17:52:32.273398 Slept 30 Ended at: 2022-05-31 17:53:02.303463',
 'Started at: 2022-05-31 17:52:42.297340 Slept 40 Ended at: 2022-05-31 17:53:22.337396',
 'Started at: 2022-05-31 17:52:52.285312 Slept 1 Ended at: 2022-05-31 17:52:53.286370',
 'Started at: 2022-05-31 17:52:53.289760 Slept 2 Ended at: 2022-05-31 17:52:55.291821',
 'Started at: 2022-05-31 17:52:55.295365 Slept 3 Ended at: 2022-05-31 17:52:58.298415',
 'Started at: 2022-05-31 17:52:58.301646 Slept 4 Ended at: 2022-05-31 17:53:02.303464',
 'Started at: 2022-05-31 17:53:02.307076 Slept 10 Ended at: 2022-05-31 17:53:12.317141',
 'Started at: 2022-05-31 17:53:02.307284 Slept 10 Ended at: 2022-05-31 17:53:12.317337',
 'Started at: 2022-05-31 17:53:12.320653 Slept 10 Ended at: 2022-05-31 17:53:22.330728']

As expected - everything starts at the same time and ends in 10, 20, and 30 seconds after each worker starts.

However, the output from the second block does not display interactive progress. It only displays this:

Starting 3 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

  0%|          | 0/3 [00:00<?, ?engine/s]

work:   0%|          | 0/23 [00:00<?, ?tasks/s]

Stopping engine(s): 1654019466
engine set stopped 1654019466: {'engines': {'0': {'exit_code': 0, 'pid': 22707, 'identifier': '0'}, '1': {'exit_code': 0, 'pid': 22709, 'identifier': '1'}, '2': {'exit_code': 0, 'pid': 22713, 'identifier': '2'}}, 'exit_code': 0}
Stopping controller
Controller stopped: {'exit_code': 0, 'pid': 22649, 'identifier': 'ipcontroller-1654019465-rj2q-13840'}

I expected the ASCII progress bars to update as progress is made. Even when I increase the sleep times to larger values (multiple minute), it doesn't update.

How can I get the progress to update as tasks complete?

How current is your JupyterLab installation? [Here](https://ipyparallel.readthedocs.io/en/latest/) says you need at least JupyterLab 3. Plus some of the support for JupyterLab was only added last year, see [here](https://github.com/ipython/ipyparallel/issues/395). Is your version of ipyparallel current? You versions of ipywidgets? (I ask since that one issue was using ipywidgets.) — Wayne, Jun 01 '22 at 20:57

score 0 · Answer 1 · answered Jun 01 '22 at 21:02

Since I cannot put well-formatted code in comments, I'll add here the following works for me to show updating progress bars in sessions launched from here (currently JupyterLab Version 3.2.8) after I run %pip install ipyparallel in the previous cell:

import time
import numpy as np
import ipyparallel as ipp
with ipp.Cluster(n=3) as rc:
    inputs = 1 + 0.1 *np.arange(10)
    view = rc.load_balanced_view()
    ar = view.map_async(time.sleep, inputs)
    ar.wait_interactive()

Code is based on the example here.

To launch an temporary session go here and press the launch binder button. A session will spin up for you after a few seconds.

I note that when I ran %pip install ipyparallel it installed tqdm as well, and so that is another package where possibility the version you have isn't quite compatible with the other packages & JupyterLab as well.

How do I get ipyparallel to print status updates while in wait_interactive() in Jupyterlab?

1 Answers1