JupyterHub Stop container/server after 5 mins of no user activity

JupyterHub
, , , ,
1

Hi folks so I have a JupyterHub which spawns docker container for each user . These containers run some ML workloads which uses GPU from the host. The problem I face is , people don’t stop server after using it. Hence some container continue to hold some chunk of GPU memory. Hence other users are deprived of GPU. Is there a feature where we can stop container/server after 5 mins of no user activity ?

Below is my config.

import dockerspawner
import os
from jupyter_client.localinterfaces import public_ips
import subprocess

c = get_config()  # noqa

c.Authenticator.admin_users = {'ross'}

c.DockerSpawner.allowed_images = {
   
    'Python TensorFlow Notebook':'quay.io/jupyter/tensorflow-notebook:latest',
    'Python Pytorch Notebook':'quay.io/jupyter/pytorch-notebook:latest'

}

# we need the hub to listen on all ips when it is in a container

c.JupyterHub.spawner_class = dockerspawner.DockerSpawner

# The docker instances need access to the Hub, so the default loopback port doesn't work:

c.JupyterHub.hub_ip = public_ips()[0]


c.JupyterHub.db_url = '/etc/jupyterhub_workspace/jupyterhub.sqlite'

c.JupyterHub.cookie_secret_file = '/etc/jupyterhub_workspace/jupyterhub_cookie_secret'

def create_dir_hook(spawner):
    username = spawner.user.name
    volume_path = os.path.join('/home/jupyter', username)
    print(volume_path)
    if not os.path.exists(volume_path):
        print("Path doesnt exist")
        subprocess.call(["/sbin/mkhomedir_helper", spawner.user.name])


c.Spawner.pre_spawn_hook = create_dir_hook

c.Spawner.default_url = '/lab'

notebook_dir = '/home/jovyan/work'

host_dir = '/home/jupyter/{username}/'
print(notebook_dir)
print(host_dir)
c.DockerSpawner.notebook_dir = notebook_dir

# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
c.DockerSpawner.volumes = {
    f'{host_dir}': notebook_dir

    }


# delete containers when the stop
c.DockerSpawner.remove = True

c.DockerSpawner.extra_create_kwargs = {'user': 'root'}

c.DockerSpawner.extra_host_config = {'runtime': 'nvidia'}

c.DockerSpawner.environment = {
    'GRANT_SUDO': 'yes',
    'CHOWN_HOME': 'yes',
    'CHOWN_EXTRA': '/home/jovyan',
    'CHOWN_HOME_OPTS': '-R',
    'NB_UID': 1000,
    'NB_GID': 1000,
    }

cc: @manics @mahendrapaipuri @consideRatio

2

JupyterHub Idle Culler is what you are looking for. You can run it as Hub Managed Service with the timeout=300 for your need.

1 Like