3

I wanted to run jupyter notebook on a node of our cluster, NOT on the login node. I could remotely run the jupyter notebook on the login node, but it would unnecessarily slow down the cluster usage. Please guide me how I can start the jupyter notebook on a node from my local desktop. Our cluster uses PBS job submission method, e.g. use use the commands like qsub, qstat, qdel to manage our jobs on the cluster.

deltasata
  • 377
  • 1
  • 4
  • 21
  • I am not sure if my cluster permits interactive node usage. But I can ssh to a node (say node1) and start jupyter notebook there. The problem in this method is that the node is not blocked and anyone can submit a job on that node. Following the the link https://dev.to/rohitfarmer/how-to-run-jupyter-notebook-in-an-interactive-node-on-a-high-performance-computer-hpc-27mg how can I claim a node on my cluster, which follows PBS job submission, to use it in interactive mode? – deltasata Jan 30 '20 at 01:42
  • It is a bit difficult to reach the person who manages our cluster right now. – deltasata Jan 30 '20 at 01:45

1 Answers1

1

You could do this by running a jupyter notebook in a jobscript. The output of the jupyter startup script can be written to a file to get the token and monitor the logs.

An example script which worked in my system was:

#!/bin/bash
#PBS -N Notebook
#PBS -j oe
#PBS -l select=1:ncpus=24
#PBS -l walltime=24:00:00
#PBS -q medium

cd $PBS_O_WORKDIR
PYTHONPATH=/path/to/python
NOTEBOOK_LOGFILE=jupyterlog.out

$PYTHONPATH/bin/jupyter notebook --no-browser --ip=0.0.0.0 --port=8890 >> ${NOTEBOOK_LOGFILE} 2>&1

You could monitor the output by tail -f jupyterlog.out and get the token id.

You could then forward and listen to the port running the jupyter notebook in your local system by,

ssh -N -f -L localhost:8888:node:8890 user@host
rashid
  • 644
  • 7
  • 18
  • What would be the python path here? – lsr729 Jun 22 '20 at 19:22
  • @lsr729 It's the path of the directory where jupyter is installed, ie, the directory which contains the bin directory containing jupyter executable. An example would be, if jupyter executable path is `/usr/bin/jupyter`, then `PYTHONPATH=/usr` – rashid Jun 23 '20 at 01:04
  • On the cluster, the jupyter file is located at `/home/lsr/.local/bin/jupyter`. So in the 3rd from the last line do I need to put `PYTHONPATH=/home/lsr/.local/bin/jupyter` ? – lsr729 Jun 23 '20 at 03:57
  • @lsr729 No it should be `PYTHONPATH=/home/lsr/.local`. Because in the last line we are including the rest of the path as `$PYTHONPATH/bin/jupyter`. – rashid Jun 23 '20 at 03:59
  • This worked, I got the password for jupyter. But then when tried to run the `ssh -N -f -L localhost:8888:node:8890 user@host` on my local computer, got the error: `bind [127.0.0.1]:8888: Permission denied channel_setup_fwd_listener_tcpip: cannot listen to port: 8888 Could not request local forwarding.` – lsr729 Jun 23 '20 at 04:06
  • I think it should be `localhost8890` instead of `localhost8888` – lsr729 Jun 23 '20 at 04:20
  • @lsr729 Did you replace `user@host` with the corresponding username and hostname of your cluster. – rashid Jun 23 '20 at 04:29
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/216463/discussion-between-lsr729-and-rashid). – lsr729 Jun 23 '20 at 04:34