We are trying to setup JupyterHub + Batchspawner on a Slurm cluster.
So far we separatly installed jupyterhub, jupyterlab notebook and batchspawner on the login and compute nodes. We manually start the jupyterhub server on the login node by running: jupyterhub --debug -f jupyterhub_config.py
jupyterhub_config.py:
c = get_config()
c.Authenticator.allow_all = True
c.JupyterHub.hub_ip = 'IP_OF_LOGINNODE_WITH_HUB'
c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
## please note, we took the default script added some debug output and had to provide an absolute batchspawner-singeluser path (otherwise command not found)
c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH --output=/sf/home/user1/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH --chdir=/sf/home/user1
#SBATCH --export=JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,USER,HOME,SHELL
#SBATCH --get-user-env=L
set -euo pipefail
trap 'echo SIGTERM received' TERM
which batchspawner-singleuser
echo $PATH
echo "ENVIRONMENT ============================"
printenv
echo "ENVIRONMENT END ========================"
echo " .. run cmd '/usr/local/bin/batchspawner-singleuser jupyterhub-singleuser'"
srun /usr/local/bin/batchspawner-singleuser jupyterhub-singleuser
echo "jupyterhub-singleuser ended gracefully"
'''
import batchspawner # Even though not used, needed to register batchspawner interface
-
Problem: “srun batchspawner-singleuser jupyterhub-singleuser” leads to “slurmstepd: error: execve(): batchspawner-singleuser: No such file or directory” on the compute node, even though the “which batchspawner-singleuser” in the batch_script returns the correct path. Using the absolute path “srun /usr/local/bin/batchspawner-singleuser jupyterhub-singleuser” in the batch_script works. Why is the command not found on the compute node? (please note, that it is there when I login as user on the compute node)
-
Problem: We can login to the JupyterHub and launch a server, however spawning it fails with following error on the compute node:
/usr/local/bin/batchspawner-singleuser # is the correct path
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ENVIRONMENT ============================
SHELL=/bin/bash
SLURM_JOB_USER=user1
SLURM_TASKS_PER_NODE=1
SLURM_JOB_UID=43643
SLURM_EXPORT_ENV=JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,USER,HOME,SHELL
SLURM_TASK_PID=159964
JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/user1/
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/home/ubuntu/miniconda3
HOSTNAME=c1-n00
SLURMD_NODENAME=c1-n00
JUPYTERHUB_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SLURM_JOB_START_TIME=1737464749
HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_PUBLIC_URL=
SLURM_CLUSTER_NAME=c1
SLURM_JOB_END_TIME=1769000749
SLURM_CPUS_ON_NODE=1
SLURM_JOB_CPUS_PER_NODE=1
PRTE_MCA_plm_slurm_args=--external-launcher
PWD=/sf/home/user1
SLURM_GTIDS=0
LOGNAME=user1
SLURM_JOB_PARTITION=global
SLURM_JOB_NUM_NODES=1
SLURM_JOBID=153
JUPYTERHUB_SERVER_NAME=
SLURM_JOB_QOS=normal
I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
HOME=/sf/home/user1
LANG=C.UTF-8
SLURM_PROCID=0
JPY_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TMPDIR=/tmp
JUPYTERHUB_SERVICE_PREFIX=/user/user1/
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/user1/oauth_callback
SLURM_TOPOLOGY_ADDR=c1-n00
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
HYDRA_BOOTSTRAP=slurm
SLURM_TOPOLOGY_ADDR_PATTERN=node
ZES_ENABLE_SYSMAN=1
USER=user1
SLURM_NODELIST=c1-n00
ENVIRONMENT=BATCH
SLURM_PRIO_PROCESS=0
JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=user1/", "access:servers!user=user1"]
JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
SHLVL=2
SLURM_NNODES=1
JUPYTERHUB_API_URL=http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/api
JUPYTERHUB_CLIENT_ID=jupyterhub-user-user1
JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=user1/", "access:servers!user=user1"]
JUPYTERHUB_PUBLIC_HUB_URL=
SLURM_SUBMIT_HOST=c1-l00
JUPYTERHUB_HOST=
SLURM_JOB_ID=153
SLURM_NODEID=0
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
SLURM_CONF=/etc/slurm/slurm.conf
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SLURM_JOB_NAME=spawner-jupyterhub
JUPYTERHUB_USER=user1
MAIL=/var/mail/user1
JUPYTERHUB_ACTIVITY_URL=http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/api/users/user1/activity
OMPI_MCA_plm_slurm_args=--external-launcher
SLURM_JOB_GID=10513
SLURM_GET_USER_ENV=1
SLURM_JOB_NODELIST=c1-n00
I_MPI_HYDRA_BOOTSTRAP=slurm
_=/usr/bin/printenv
ENVIRONMENT END ========================
.. run cmd '/usr/local/bin/batchspawner-singleuser jupyterhub-singleuser'
Traceback (most recent call last):
File "/usr/lib/python3.10/pkgutil.py", line 417, in get_importer
importer = sys.path_importer_cache[path_item]
KeyError: None
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/batchspawner-singleuser", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/batchspawner/singleuser.py", line 47, in main
run_path(cmd_path, run_name="__main__")
File "/usr/lib/python3.10/runpy.py", line 279, in run_path
importer = get_importer(path_name)
File "/usr/lib/python3.10/pkgutil.py", line 421, in get_importer
importer = path_hook(path_item)
File "<frozen zipimport>", line 67, in __init__
File "/usr/lib/python3.10/os.py", line 823, in fsdecode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
srun: error: c1-n00: task 0: Exited with exit code 1
It seems that some path_item is missing, maybe some required environment variable is not set?
Did we maybe forgot to set some required config parameter?
JupyterHub server log
[D 2025-01-21 14:05:33.860 JupyterHub application:908] Looking for jupyterhub_config in /home/ubuntu/miniconda3
[D 2025-01-21 14:05:33.872 JupyterHub application:929] Loaded config file: /home/ubuntu/miniconda3/jupyterhub_config.py
[I 2025-01-21 14:05:33.882 JupyterHub app:3346] Running JupyterHub version 5.2.1
[I 2025-01-21 14:05:33.882 JupyterHub app:3376] Using Authenticator: jupyterhub.auth.PAMAuthenticator-5.2.1
[I 2025-01-21 14:05:33.882 JupyterHub app:3376] Using Spawner: batchspawner.batchspawner.SlurmSpawner-1.3.0
[I 2025-01-21 14:05:33.882 JupyterHub app:3376] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.1
/home/ubuntu/miniconda3/envs/jupyterhub/lib/python3.13/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
validate_schema(_schema)
[I 2025-01-21 14:05:33.888 JupyterHub app:1831] Loading cookie_secret from /home/ubuntu/miniconda3/jupyterhub_cookie_secret
[D 2025-01-21 14:05:33.889 JupyterHub app:1998] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2025-01-21 14:05:33.911 JupyterHub orm:1509] database schema version found: 4621fec11365
[I 2025-01-21 14:05:33.964 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2025-01-21 14:05:33.965 JupyterHub app:2338] Loading roles into database
[D 2025-01-21 14:05:33.989 JupyterHub app:2685] Purging expired APITokens
[D 2025-01-21 14:05:33.992 JupyterHub app:2685] Purging expired OAuthCodes
[D 2025-01-21 14:05:33.994 JupyterHub app:2685] Purging expired Shares
[D 2025-01-21 14:05:33.995 JupyterHub app:2685] Purging expired ShareCodes
[D 2025-01-21 14:05:33.996 JupyterHub app:2459] Loading role assignments from config
[D 2025-01-21 14:05:34.010 JupyterHub app:2970] Initializing spawners
[D 2025-01-21 14:05:34.018 JupyterHub app:3120] Loaded users:
[I 2025-01-21 14:05:34.018 JupyterHub app:3416] Initialized 0 spawners in 0.008 seconds
[I 2025-01-21 14:05:34.022 JupyterHub metrics:373] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-01-21 14:05:34.023 JupyterHub metrics:373] Found 3 active users in the last ActiveUserPeriods.seven_days
[I 2025-01-21 14:05:34.024 JupyterHub metrics:373] Found 3 active users in the last ActiveUserPeriods.thirty_days
[W 2025-01-21 14:05:34.024 JupyterHub proxy:748] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2025-01-21 14:05:34.024 JupyterHub proxy:752] Starting proxy @ http://:8000
[D 2025-01-21 14:05:34.024 JupyterHub proxy:753] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/error', '--log-level', 'info']
[D 2025-01-21 14:05:34.025 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
[D 2025-01-21 14:05:34.026 JupyterHub utils:272] Waiting 10s for server at c1-l00:8000
[D 2025-01-21 14:05:34.026 JupyterHub utils:119] Server at c1-l00:8000 not ready: [Errno 111] Connection refused
[D 2025-01-21 14:05:34.026 JupyterHub utils:272] Waiting 10s for server at 127.0.0.1:8001
[D 2025-01-21 14:05:34.026 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2025-01-21 14:05:34.077 JupyterHub utils:119] Server at c1-l00:8000 not ready: [Errno 111] Connection refused
[D 2025-01-21 14:05:34.147 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2025-01-21 14:05:34.251 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
14:05:34.287 [ConfigProxy] info: Proxying http://*:8000 to (no default)
14:05:34.290 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2025-01-21 14:05:34.314 JupyterHub utils:280] Server at c1-l00:8000 responded in 0.29s
[D 2025-01-21 14:05:34.900 JupyterHub utils:280] Server at 127.0.0.1:8001 responded in 0.87s
[D 2025-01-21 14:05:34.900 JupyterHub proxy:832] Proxy started and appears to be up
[D 2025-01-21 14:05:34.901 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2025-01-21 14:05:34.909 JupyterHub app:3739] Hub API listening on http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/
[D 2025-01-21 14:05:34.909 JupyterHub proxy:389] Fetching routes to check
[D 2025-01-21 14:05:34.909 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:05:34.909 [ConfigProxy] info: 200 GET /api/routes
14:05:34.911 [ConfigProxy] info: 200 GET /api/routes
[D 2025-01-21 14:05:34.911 JupyterHub proxy:392] Checking routes
[I 2025-01-21 14:05:34.911 JupyterHub proxy:477] Adding route for Hub: / => http://IP_OF_LOGINNODE_WITH_HUB:8081
[D 2025-01-21 14:05:34.912 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
14:05:34.914 [ConfigProxy] info: Adding route / -> http://IP_OF_LOGINNODE_WITH_HUB:8081
14:05:34.914 [ConfigProxy] info: Route added / -> http://IP_OF_LOGINNODE_WITH_HUB:8081
[I 2025-01-21 14:05:34.915 JupyterHub app:3770] JupyterHub is now running at http://:8000
14:05:34.915 [ConfigProxy] info: 201 POST /api/routes/
[D 2025-01-21 14:05:34.916 JupyterHub app:3339] It took 1.060 seconds for the Hub to start
(node:143266) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Please use Object.assign() instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[I 2025-01-21 14:05:44.655 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:xxx.xxx.xx.93) 1.03ms
[D 2025-01-21 14:05:44.700 JupyterHub base:411] Refreshing auth for user1
[D 2025-01-21 14:05:44.700 JupyterHub user:496] Creating <class 'batchspawner.batchspawner.SlurmSpawner'> for user1:
[I 2025-01-21 14:05:44.701 JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (user1@::ffff:xxx.xxx.xx.93) 18.77ms
[D 2025-01-21 14:05:44.716 JupyterHub scopes:1010] Checking access to /hub/spawn via scope servers!server=user1/
[D 2025-01-21 14:05:44.717 JupyterHub pages:216] Triggering spawn with default options for user1
[D 2025-01-21 14:05:44.717 JupyterHub base:1095] Initiating spawn for user1
[D 2025-01-21 14:05:44.717 JupyterHub base:1099] 0/100 concurrent spawns
[D 2025-01-21 14:05:44.717 JupyterHub base:1104] 0 active servers
[I 2025-01-21 14:05:44.763 JupyterHub provider:661] Creating oauth client jupyterhub-user-user1
[D 2025-01-21 14:05:44.791 JupyterHub user:913] Calling Spawner.start for user1
[I 2025-01-21 14:05:44.793 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cluster': '', 'epilogue': '', 'gres': '', 'homedir': '/sf/home/user1', 'host': '', 'keepvars': 'JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,USER,HOME,SHELL', 'keepvars_extra': '', 'memory': '', 'ngpus': '', 'nprocs': '', 'options': '', 'partition': '', 'prologue': '', 'qos': '', 'queue': '', 'reservation': '', 'runtime': '', 'srun': 'srun', 'username': 'user1', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser'}
[I 2025-01-21 14:05:44.793 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u user1 sbatch --parsable
[D 2025-01-21 14:05:44.793 JupyterHub batchspawner:283] Spawner submitting script:
#!/bin/bash
#SBATCH --output=/sf/home/user1/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH --chdir=/sf/home/user1
#SBATCH --export=JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,USER,HOME,SHELL
#SBATCH --get-user-env=L
set -euo pipefail
trap 'echo SIGTERM received' TERM
which batchspawner-singleuser
echo $PATH
echo "ENVIRONMENT ============================"
printenv
echo "ENVIRONMENT END ========================"
echo " .. run cmd '/usr/local/bin/batchspawner-singleuser jupyterhub-singleuser'"
srun /usr/local/bin/batchspawner-singleuser jupyterhub-singleuser
echo "jupyterhub-singleuser ended gracefully"
[D 2025-01-21 14:05:44.794 JupyterHub batchspawner:284] Spawner submitting environment: {'JUPYTERHUB_API_TOKEN': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'JPY_API_TOKEN': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-user1', 'JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED': '0', 'JUPYTERHUB_HOST': '', 'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/user1/oauth_callback', 'JUPYTERHUB_OAUTH_SCOPES': '["access:servers!server=user1/", "access:servers!user=user1"]', 'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!server=user1/", "access:servers!user=user1"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]', 'JUPYTERHUB_USER': 'user1', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL': 'http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://IP_OF_LOGINNODE_WITH_HUB:8081/hub/api/users/user1/activity', 'JUPYTERHUB_BASE_URL': '/', 'JUPYTERHUB_SERVICE_PREFIX': '/user/user1/', 'JUPYTERHUB_SERVICE_URL': 'http://0.0.0.0:0/user/user1/', 'JUPYTERHUB_PUBLIC_URL': '', 'JUPYTERHUB_PUBLIC_HUB_URL': '', 'USER': 'user1', 'HOME': '/sf/home/user1', 'SHELL': '/bin/bash'}
[W 2025-01-21 14:05:45.717 JupyterHub base:201] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I 2025-01-21 14:05:45.732 JupyterHub log:192] 302 GET /hub/spawn -> /hub/spawn-pending/user1 (user1@::ffff:xxx.xxx.xx.93) 1007.40ms
[D 2025-01-21 14:05:45.771 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/user1 via scope servers!server=user1/
[I 2025-01-21 14:05:45.774 JupyterHub pages:397] user1 is pending spawn
[I 2025-01-21 14:05:45.799 JupyterHub log:192] 200 GET /hub/spawn-pending/user1 (user1@::ffff:xxx.xxx.xx.93) 32.61ms
[W 2025-01-21 14:05:45.953 JupyterHub _xsrf_utils:195] Skipping XSRF check for insecure request GET /hub/api/users/user1/server/progress
[D 2025-01-21 14:05:45.953 JupyterHub scopes:1010] Checking access to /hub/api/users/user1/server/progress via scope read:servers!server=user1/
[I 2025-01-21 14:05:49.769 JupyterHub batchspawner:287] Job submitted. output: 153
[D 2025-01-21 14:05:49.770 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u user1 squeue -h -j 153 -o '%T %B'
[D 2025-01-21 14:05:49.833 JupyterHub batchspawner:428] Job 153 still pending
[D 2025-01-21 14:05:50.334 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u user1 squeue -h -j 153 -o '%T %B'
[W 2025-01-21 14:05:54.718 JupyterHub base:1227] User user1 is slow to start (timeout=10)
[D 2025-01-21 14:06:01.819 JupyterHub base:366] Recording first activity for <APIToken('xxxx...', user='user1', client_id='jupyterhub')>
[I 2025-01-21 14:06:01.830 JupyterHub log:192] 200 POST /hub/api/batchspawner (user1@IP_OF_LOGINNODE_WITH_HUB) 13.59ms
[W 2025-01-21 14:06:02.065 JupyterHub spawner:179]
The shared database session at Spawner.db is deprecated, and will be removed.
Please manage your own database and connections.
Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
if you have questions or ideas about direct database needs for your Spawner.
[I 2025-01-21 14:06:02.065 JupyterHub batchspawner:455] Notebook server job 153 started at c1-n00:52335
[D 2025-01-21 14:06:02.074 JupyterHub spawner:1475] Polling subprocess every 30s
[D 2025-01-21 14:06:02.083 JupyterHub utils:292] Waiting 30s for server at http://c1-n00:52335/user/user1/api
[W 2025-01-21 14:06:30.366 JupyterHub user:1055] user1's server never showed up at http://c1-n00:52335/user/user1/ after 30 seconds. Giving up.
Common causes of this timeout, and debugging tips:
1. The server didn't finish starting,
or it crashed due to a configuration issue.
Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
This may be a configuration issue specific to your chosen Spawner.
Check the single-user server logs and resource to make sure the URL
is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
To fix: increase `Spawner.http_timeout` configuration
to a number of seconds that is enough for servers to become responsive.
[D 2025-01-21 14:06:30.366 JupyterHub user:1100] Stopping user1
[D 2025-01-21 14:06:30.366 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u user1 squeue -h -j 153 -o '%T %B'
[D 2025-01-21 14:06:30.443 JupyterHub user:1122] Deleting oauth client jupyterhub-user-user1
[D 2025-01-21 14:06:30.469 JupyterHub user:1125] Finished stopping user1
[E 2025-01-21 14:06:30.483 JupyterHub gen:629] Exception in Future <Task finished name='Task-29' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /home/ubuntu/miniconda3/envs/jupyterhub/lib/python3.13/site-packages/jupyterhub/handlers/base.py:1115> exception=TimeoutError("Server at http://c1-n00:52335/user/user1/api didn't respond in 30 seconds")> after timeout
Edit:
Some new finding, I tried to debug “singleuser.py”, this is the file where run_path statement fails. I put following prints there:
One can see that which(…) does return None. I think fixing problem 1) will also fix problem 2).
sys.argv[1]:
jupyterhub-singleuser
cmd_path:
None