8

I'm forced to work with JupyterLab on a Linux-server which I do not host myself. The problem is that the Jupyter-process takes a lot of memory; this has been part in several bug reports like here and here.

Anyhow, as can be anticipated from the introduction, I do not have any sudo-rights and therefore cannot restart the lab myself (at least I think that this should not be possible for me).

Whats odd in my opinion can be seen on this Screenshot taken from HTOP:

JupyterLab

The bash command which startet the lab has a lot of subprocesses which all look like kernels I opened and closed over the entire usage-time (the server is up for a month and I opened and closed a lot of kernels; none was running when taking the picture).

Since every of these processes ends on .json, I assume that these might be some still intact runtime parameters. All processes on the third level look identical like on the screenshot, nothing else there.

Anyhow, I don't want to resolve the memory-spilling-bug. My question is rather straight forward:

Since no kernel is running: Can I just kill all of the processes on the third level and free the memory by doing so, or might this crash the lab?
Not crashing the lab is essential since I have no possibility to restart the lab.

Markus
  • 2,265
  • 5
  • 28
  • 54
  • 2
    did you ever find the root cause of this and a solution - rather than a killer workaround? – jtlz2 Sep 07 '20 at 08:17
  • 1
    @jtlz2: Unfortunately not. I have to admit that I just accepted the workaround and some time later contacted the admin for a restart of the system which also helped. – Markus Sep 07 '20 at 08:42

2 Answers2

2

Luckily, I stumbled upon this website here. Here the writer has a related problem with unreleased GPU-memory and also a lot of ipykernel_launchers.

He solved this problem by using the proposed drastic way.

  1. First, display all your ipykernel_launchers with ps -aux|grep ipykernel_launcher (analog to the HTOP-screenshot in the question).
  2. Kill every process one by one with kill -9 PID1 PID2 .... Be sure to only kill everything in the style of (nothing else):

    /usr/bin/python3 -m ipykernel_launcher -f /content/.local/share/jupyter/runtime/kernel-95cb65b9-23eb-4f87-801b-d995ca30fc32.json

  3. Memory should be freed now and JupyterLab should be still running.

Edit:

Thanks to @EightBitTony on Unix&Linux-StackExchange I now know that there is a command to kill all processes in one step:

for pid in $(ps -ef | grep -v "awk" | awk '/ipykernel_launcher/ {print $2}'); do kill -9 $pid; done

Please be aware what you do when you do it this way!

Markus
  • 2,265
  • 5
  • 28
  • 54
1

There's actually a built in way to kill processes given a certain pattern:

pkill -f /<regex-pattern>/

In your case, pkill -f ".*/ipykernel_launcher/.*" would do the trick

Paul Roub
  • 36,322
  • 27
  • 84
  • 93
Leitouran
  • 578
  • 3
  • 16