0

I have notebooks hosted on a remote server which take a long time to load due to the amount of data being moved into memory. These notebooks are accessed via my employer's VPN and when my machine becomes idle, the VPN connection drops and my notebook loses connection to the remote server. Reloading the data takes a significant amount of time. Is it possible to automatically reconnect the notebook in the browser to the remote server after disconnection? Alternatively, are there any other ways to mitigate this issue? This situation causes a significant disruption in my workflow as I may need to rerun a cell which takes 40 minutes to execute, only to lose my VPN connection.

user1247336
  • 1
  • 1
  • A related question is when my notebooks are disconnected and I attempt to run a cell, it shows [*] to the left of the cell. I've spent long time waiting for a cell to run, only to find out it is not even connected to the server. – user1247336 Jun 16 '23 at 15:15
  • Can you use IPython console/interpreter to do this? Maybe in conjunction with screen or tmux? (I haven't checked recently how robust such an idea might be in your situation.) There are things where Jupyter isn't the best tool to be using. Generally long and costly computation can be one of them. See [here](https://discourse.jupyter.org/t/auto-save-periodically-during-nbconvert/17361/3?u=fomightez). It is also possible to use Jupytext or Papermill to run notebooks on the command line along with screen or tmux. That may be something to look into. – Wayne Jun 16 '23 at 16:06
  • @Wayne thanks for your comment. Can you briefly describe the workflow (and the benefit) using the python console vs Jupyter? – user1247336 Jun 17 '23 at 14:59
  • I don't what else you expect me to say? Jupyter has a lot of overhead and choices of how it works that for long computation it can be an issue **as you've experienced yourself so that more traditional ways of running processes can be better** . Others put it more clear than me, see [here](https://news.ycombinator.com/item?id=19859913) and [here](https://github.com/jupyter/notebook/issues/641) and [here](https://saturncloud.io/blog/long-running-notebooks/) and [here](https://medium.com/codex/an-honest-rant-on-why-i-despise-jupyter-notebooks-6b631334ce19). – Wayne Jun 17 '23 at 16:00

0 Answers0