Running mulitple jupyterhub instances on single system (different ports)

JupyterHub discuss
1

Hi. I’m very new to jupyterhub and am curious about whether or not a configuration I’m trying is even possible.

Our environment is a single slurm compute cluster. There are several different projects that use the cluster and each projects data is separated into a different NFS share. We’d like to set up Jupyterhub to be usable by different projects, but have each projects area separate from each other.

My thought was to run multiple jupyterhub instances…one for each project. Those instances would have a slightly different configuration, like a different base URL jhub.example.org/projA, jhub.example.org/projB, etc…

I guess question 1, is that a reasonable thing to attempt to do?

I have jupyterhub working right now using the default ports (8000,8081, and 8001 for the proxy). I set up an Apache reverse proxy to send to JupyterHub, and the job submission works for sending to slurm (using batchspawner/wrapspawner), the notebook server starts and everything is fine.

If I attempt to change the ports that the JupyterHub system uses, that’s when I start running in to issues.

I set the following options (I can provide more details if needed):
c.JupyerHub.bind_url = ‘127.0.0.1:8100/projA’ (there is an http in front of this)
c.JupyterHub.hub_port = 8181
c.ConfigurableHTTPProxy.api_url = ‘127.0.0.1:8101’

With those changes, I can start up a Jupyerhub server, and it does come up, and I can log into it. I then select my option from wrapspawner, slurm sees the job go through and does start execution of batchspawner-singleuser on the compute node, but at that point, I get this error message logged from slurm:

Traceback (most recent call last):
File “/usr/lib64/python3.6/pkgutil.py”, line 412, in get_importer
importer = sys.path_importer_cache[path_item]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/jupyter/bin/batchspawner-singleuser”, line 6, in
main()
File “/usr/local/jupyter/lib64/python3.6/site-packages/batchspawner/singleuser.py”, line 22, in main
run_path(cmd_path, run_name=“main”)
File “/usr/lib64/python3.6/runpy.py”, line 252, in run_path
importer = get_importer(path_name)
File “/usr/lib64/python3.6/pkgutil.py”, line 416, in get_importer
importer = path_hook(path_item)
TypeError: expected str, bytes or os.PathLike object, not NoneType

If I set the ports back to default, it all works. I’m not really sure what I’m missing.

Thanks for any help that can be offered.

2

It should work. You’ll need to run the two JupyterHubs from different working directories, and ensure you use a remote spawner which you are.

Since JupyterHub successfully starts it sounds like the problem may be in BatchSpawner. I’m not familiar with it, but if you share your full configuration with secrets redacted someone maybe able to help.

3

I agree that it is likely something with batchspawner, but here is the configuration file (comments removed) of the jupyterhub instance that I have running.

c.JupyterHub.bind_url = ‘http://127.0.0.1:8100/projA’
c.JupyterHub.cookie_secret_file = ‘/usr/local/etc/jhub-projA_cookie_secret’
c.JupyterHub.extra_handlers = [(r"/api/batchspawner", ‘batchspawner.api.BatchSpawnerAPIHandler’)]
c.JupyterHub.hub_connect_ip = ‘XX.XX.XX.XX’
c.JupyterHub.hub_ip = ‘XX.XX.XX.XX’
c.JupyterHub.hub_port = 8181
c.JupyterHub.pid_file = ‘/var/tmp/jhub-projA.pid’
c.Spawner.cmd = [’/usr/local/bin/jupyter-labhub’]
c.Authenticator.allowed_groups = {‘projA-grouplist’}
c.Authenticator.create_system_users = False
c = get_config()
c.JupyterHub.spawner_class = ‘batchspawner.SlurmSpawner’
c.BatchSpawnerBase.exec_prefix = “sudo -E -u {username} -g projA”
c.Spawner.http_timeout = 240
c.SlurmSpawner.req_nprocs = ‘1’
c.SlurmSpawner.req_runtime = ‘1:00:00’
c.SlurmSpawner.req_partition = ‘cluster’
c.SlurmSpawner.start_timeout = 240
c.SlurmSpawner.batch_script = ‘’’#!/bin/bash

export PROJECTDIR=/exports/projA
if [ ! -d “$PROJECTDIR/jupyter” ]
then
mkdir $PROJECTDIR/jupyter
fi

if [ ! -d “$PROJECTDIR/jupyter/{username}” ]
then
mkdir $PROJECTDIR/jupyter/{username}
fi

export HOME=$PROJECTDIR/jupyter/{username}

source /software/global/jupyter/bin/activate

{cmd}
‘’’

c.JupyterHub.spawner_class = ‘wrapspawner.ProfilesSpawner’

c.ProfilesSpawner.profiles = [
(“Local server: (Run on local machine)”, “local”,
“jupyterhub.spawner.LocalProcessSpawner”, {‘ip’:‘0.0.0.0’}),
(“Single CPU: (1 CPU, 8GB, 1 hr)”, “cpu1”, “batchspawner.SlurmSpawner”,
dict(req_options=" -n 1 -t 1:00:00 -p cluster --mem=8G “)),
(“Single GPU: (1 CPU, 1 GPU, 8GB, 1 hr)”, “gpu1”,
“batchspawner.SlurmSpawner”,
dict(req_options=” -n 1 -t 1:00:00 -p gpu --mem=8G --gres=gpu:1")),
]

c.ConfigurableHTTPProxy.api_url = ‘http://127.0.0.1:8101’
c.ConfigurableHTTPProxy.pid_file = ‘/var/tmp/jhub-projA-proxy.pid’