How to close JupyterHub session/kernell programmatically

JupyterHub
1

Hi.

I want to write a script that would make sure that all other sessions/kernels on the server are currently down. Because if they all are active, my server runs out of RAM. How do I do it? I was thinking about bash script that would run something like:
jupyter notebook list
jupyter notebook stop 8888

But this only works for jupyter. And what about JupyterHub? If I run “jupyter notebook list” I get:
Currently running servers:
http://127.0.0.1:46103/user/ubuntu/ :: /home/ubuntu
Instead of list of sessions. If I run “jupyterhub notebook list”, I get error:

[E 2021-04-26 19:00:16.966 JupyterHub app:2341] Failed to bind hub to http://127.0.0.1:8081/hub/
[E 2021-04-26 19:00:16.966 JupyterHub app:2482]
    Traceback (most recent call last):
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/jupyterhub/app.py", line 2480, in launch_instance_async
        await self.start()
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/jupyterhub/app.py", line 2336, in start
        self.http_server.listen(port, address=ip)
      File "/home/ubuntu/.local/lib/python3.7/site-packages/tornado/tcpserver.py", line 151, in listen
        sockets = bind_sockets(port, address=address)
      File "/home/ubuntu/.local/lib/python3.7/site-packages/tornado/netutil.py", line 161, in bind_sockets
        sock.bind(sockaddr)
    OSError: [Errno 98] Address already in use

This is not the complete stack, but I am trying to be short. I also tried running this code in a cell:
%%javascript
Jupyter.notebook.session.delete();
But it only stops the current notebook, but I need the opposite - to stop all the other notebooks.
I also tried inserting this code in another notebook, and then run it from the first notebook using %run "notebook location" but it also stops the session I run it from rather than the one where this script is located. So please tell me how to close other sessions programmatically.

2

First, the notebook server has its own options for shutting down idle kernels, which may be sufficient for you without scripting it:

# the idle timeout (in seconds): how much time to wait before shutting down an idle kernel
c.MappingKernelManager.cull_idle_timeout = 10 * 60
# how often to check for idle kernels (seconds)
c.MappingKernelManager.cull_interval = 1 * 60
# whether to consider kernels with active connections available for culling (open in lab, etc.)
# default is False, so that leaving a tab open is a lightweight way to ensure
# a kernel doesn't get shut down.
c.MappingKernelManager.cull_connected = True

I’m not sure what do you mean by a list of sessions that’s different between jupyter and jupyterhub. jupyter notebook list lists running notebook servers and works the same in both. In JupyterHub, there will typically only be one notebook server per user. There may be more than one if your jupyterhub deployment allows named servers, but depending on your Spawner, they are likely not visible from each other and won’t be returned from jupyter notebook list. You can use the JupyterHub API to list your other running servers, though.

If you still want to do this explicitly, you can use the Jupyter REST API to list and shutdown sessions based on activity, connections, busy/idle status, etc. here is a notebook example that will run from jupyterhub and close all but the most recently active servers. In JupyterHub, there is always an API token $JUPYTERHUB_API_TOKEN that can be used to communicate with the notebook server’s API.

A session model looks like this:

{'id': '42481fdd-c6a8-4e69-ae9f-c2c2b6b2c233',
  'path': 'cull-kernels.ipynb',
  'name': 'cull-kernels.ipynb',
  'type': 'notebook',
  'kernel': {'id': '1e884864-3de1-4cad-a1f1-8f4a055888e4',
   'name': 'python3',
   'last_activity': '2021-04-27T09:41:09.657388Z',
   'execution_state': 'busy',
   'connections': 2},
  'notebook': {'path': 'cull-kernels.ipynb', 'name': 'cull-kernels.ipynb'}}

You can use the last_activity, execution_state, and/or connections fields to make your decisions about which sessions to close. This is the same information the internal culling configuration uses.

2 Likes
3

Hi, minkr. Thank you for the solution. The Rest API approach works perfectly. I actually came across it when I was trying to solve this myself, but I didn’t realize that I had to get server port and include username in the query. So I couldn’t get it work and I concluded that it only works for regular Jupyter.

I will also implement the timer approach in case the user forgets to run the explicit script.

The reason why I was trying jupyter notebook list command was that I believed that it returns the list of sessions instead of servers. I concluded it from here:
https://deeplearning.lipingyang.org/2018/03/29/checking-from-command-line-if-jupyter-server-is-running-and-kill-if-needed/
They say there “You can use the following command to kill specific notebook that you would like to stop.”
$ jupyter notebook stop 8888 I thought that by notebook they meant ipynb file (in other words session). Turns out that by notebook they meant a server.

4

So I tried adding
c.MappingKernelManager.cull_idle_timeout = 10 * 60
c.MappingKernelManager.cull_interval = 1 * 60
c.MappingKernelManager.cull_connected = True

and they don’t seem to work. I tried small intervals like 10 sec and there was no effect. jupyterhub_config.py doesn’t contain these lines by default, so it seems to me that they don’t work for Jupyterhub. Is there any other way to shut down idle kernels, or is there’s something I did wrong with the approach above? I already implemented shutting down idle server and it does work but I also want to shut idle kernels

5

And also, I hope you can advise me on how to make sure that kernel stops when the tab associated with it is closed. It seems to me that it worked like that before, but after I moved to a different server, it no longer shuts the kernel on tab close. I tried adding js script to ~/.jupyter/custom/custom.js - no effect. So what should I do to make it work for Jupyterhub?

6

Ok, now shutting down idle kernels works. Turns out I had to insert the code above into jupyter_notebook_config.py . I didn’t know that so I put it into jupyterhub_config.py first and it didn’t work. Minkr, you should’ve mentioned it at the beginning :slight_smile: