Having a dynamic drop down for profile spawner

JupyterHub
,
1

Hi, I have the following jupyterhub config:

# Configuration file for jupyterhub.

import batchspawner
import systemdspawner
c = get_config()

c.JupyterHub.hub_connect_ip = 


c.JupyterHub.hub_ip = 


c.Spawner.env_keep = ['PATH', 'PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL', 'JUPYTERHUB_SINGLEUSER_APP']


c.Spawner.start_timeout = 120

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.Spawner.cmd = ['jupyter-labhub']

c.Spawner.http_timeout = 120

c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH --output={{homedir}}/Jupyter/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=jupyterhub
#SBATCH --chdir={{homedir}}
#SBATCH --export={{keepvars}}
#SBATCH --get-user-env=L
{% if partition  %}#SBATCH --partition={{partition}}
{% endif %}{% if runtime    %}#SBATCH --time={{runtime}}
{% endif %}{% if memory     %}#SBATCH --mem={{memory}}
{% endif %}{% if gres       %}#SBATCH --gres={{gres}}
{% endif %}{% if nprocs     %}#SBATCH --cpus-per-task={{nprocs}}
{% endif %}{% if reservation%}#SBATCH --reservation={{reservation}}
{% endif %}{% if options    %}#SBATCH {{options}}{% endif %}

set -euo pipefail
trap 'echo SIGTERM received' TERM

module load git
module load jupyterhub/1.1

{{prologue}}
which jupyterhub-singleuser
{{cmd}}
echo "jupyterhub-singleuser ended gracefully"
{{epilogue}}
'''

c.SystemdSpawner.disable_user_sudo = True


c.ProfilesSpawner.ip = '0.0.0.0'
c.ProfilesSpawner.profiles = [
 
# ('Local server. Use it !*ONLY FOR DEVELOPMENT*!', 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
 
# ('Local server. Use it !*ONLY FOR DEVELOPMENT*!', 'systemdlocal', 'systemdspawner.SystemdSpawner', {'ip':'0.0.0.0'} ),
 ('Local server - Use it !*ONLY FOR DEVELOPMENT*! 16GB RAM, 8 CPUs', 'local_limited', 'systemdspawner.SystemdSpawner', {'ip':'0.0.0.0', 'mem_limit':'16G', 'cpu_limit':4.0,'disable_user_sudo': True,'cmd': ['jupyter-labhub']}),
#
 ('Cluster - 1 CPU core, 4GB RAM, No GPU, 8 hours', 'c4gb0gpu8h', 'batchspawner.SlurmSpawner', dict(req_nprocs='1', req_partition='defaultp', req_runtime='8:00:00', req_memory='4G', req_gpu='0')),
 ('Cluster - 4 CPU core, 10GB RAM, No GPU, 48 hours', '4c10gb0gpu48h', 'batchspawner.SlurmSpawner', dict(req_nprocs='4', req_partition='defaultp', req_runtime='48:00:00', req_memory='10G', req_gpu='0')),
 ('Cluster - 8 CPU core, 20GB RAM, No GPU, 48 hours', '8c20gb0gpu48h', 'batchspawner.SlurmSpawner', dict(req_nprocs='8', req_partition='defaultp', req_runtime='48:00:00', req_memory='20G', req_gpu='0')),
 ('Cluster - 16 CPU core, 32GB RAM, No GPU, 48 hours', '8c32gb0gpu48h', 'batchspawner.SlurmSpawner', dict(req_nprocs='16', req_partition='defaultp', req_runtime='48:00:00', req_memory='32G', req_gpu='0')),
 ('Cluster - 4 CPU core, 60GB RAM, No GPU, 48 hours', '4c60gb0gpu48h', 'batchspawner.SlurmSpawner', dict(req_nprocs='4', req_partition='defaultp', req_runtime='48:00:00', req_memory='60G', req_gpu='0'))

]


c.Authenticator.admin_users = {"user1"}

I want to know if there is a way to make the profiles dynamic in a sense that I don’t have to restart the server everytime I add a new profile but rather edit e.g. a json or yaml file to add or remove more profiles and they show up dynamically on the user’s end.

2

Have a look at

You can generate the profile list from a function that pulls information from somewhere else.

2 Likes
3

Real stupid question on my part but isn’t this solution valid for Kubernates spawner? As far as I could see I could not find a similar option in the batch spawner for slurm or wrap spawner but that could just be me being thick

4

Sorry, you’re right!

Both spawners follow the same principals though for customisation. wrapspawner includes an example of dynamically generating the profile list for docker images:

5

Something like the following should work for your use case:

from WrapSpawner import ProfilesSpawner

class MyProfilesSpawner(ProfilesSpawner):
    
    def __init__(self):
        super().__init__()
        try:
            with open('/path/to/profiles.json') as f:
                self.profiles = json.load(f)
        except Exception as e:
            self.log.exception('Failed to read profiles.json: %s' % e)
            
c.JupyterHub.spawner_class = MyProfilesSpawner
6

I tried a similar approach before, but the issue with that approach was that it only read the file when I first start jupyterhub, and not was similar to the static setup I had. But I will try your approach again to double check as I may have had tried something different. Will post next week with the results! Thank you

7

Hmm interesting, I will have a look at the approach and try it along with mahendrapaipuri’s approach, but in all honesty it will take a few days due to other work and me being a bit rusty (the jupyterhub was maintained by someone else before)

8

I guess the spawner object is instantiated on every spawn. In any case, if that does not work, you can overload _options_form_default method instead of __init__.

def _options_form_default(self):
        try:
            with open('/path/to/profiles.json') as f:
                self.profiles = json.load(f)
        except Exception as e:
            self.log.exception('Failed to read profiles.json: %s' % e)
        return super()._options_form_default()