Hi,
Jupyterhub is running on the login node of our Slurm cluster.
Jupyter server starts (the slurm job runs jupyterhub-singleuser in a compute node), but in the browser only “A Jupyter Server is running” appears.
Package versions:
batchspawner 1.3.1.dev0
jupyter_server 2.14.0
jupyterlab 4.1.8
notebook 7.1.3
wrapspawner 1.0.2.dev0
Thanks advanced
I tried to include details in the first message, but discourse refused it…
Next more info on our setup:
jupyterhub_config.py:
import batchspawner
c.JupyterHub.spawner_class = ‘wrapspawner.ProfilesSpawner’
c.SlurmSpawner.batchspawner_singleuser_cmd = ‘/usr/local/envs/python/jupyterhub/4.1.3/bin/batchspawner-singleuser’
singleuser_cmd = ‘/usr/local/envs/python/jupyterhub/4.1.3/bin/jupyterhub-singleuser’
slurm_default_cfg = dict(req_partition=‘’, req_options=‘’, req_workdir=‘/home/{user}’,
cmd=singleuser_cmd, req_nprocs=‘’, req_prologue=‘’, req_srun=‘’)
profiles = [
[“Test”, ‘test’, ‘batchspawner.SlurmSpawner’,
{**slurm_default_cfg, **dict(req_partition=‘test’, req_memory=‘4200’, req_runtime=‘2-0’)}],
]
c.ProfilesSpawner.profiles = profiles
Can you share the startup logs of the server?
Does it work if you set c.Spawner.default_url = "/lab"?
Hi minrk,
Sure, here it is the slurm Job log:
[I 2024-05-17 14:12:25.851 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.5
[W 2024-05-17 14:12:25.851 JupyterHubSingleUser] No default url found in config or known extensions, searching other extensions for default_url
[W 2024-05-17 14:12:25.851 JupyterHubSingleUser] Found no extension with a default URL, UI will likely be unavailable
[I 2024-05-17 14:12:25.852 ServerApp] jupyterhub | extension was successfully linked.
/usr/local/envs/python/jupyterhub/4.1.3/lib/python3.11/site-packages/jupyter_server/serverapp.py:2236: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/jupytest/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/jupytest/oauth_callback
self.web_app = ServerWebApplication(
[I 2024-05-17 14:12:25.874 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-05-17 14:12:25.874 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-05-17 14:12:25.874 ServerApp] Serving notebooks from local directory: /home/jupytest
[I 2024-05-17 14:12:25.874 ServerApp] Jupyter Server 2.14.0 is running at:
[I 2024-05-17 14:12:25.874 ServerApp] http://c01:36975/user/jupytest/?token=...
[I 2024-05-17 14:12:25.874 ServerApp] http://127.0.0.1:36975/user/jupytest/?token=...
[I 2024-05-17 14:12:25.874 ServerApp] Use Control-C to stop this server and shut down all kernel
Thanks,
(this was without using /lab, I will try c.Spawner.default_url = "/lab" and come back)
The output suggests that JupyterLab server extension is not found. In the singleuser server environment, can you share the output of:
pip listjupyter server extension listHi Minrk,
In the env /usr/local/envs/python/jupyterhub/4.1.3 (which is defined as single_user env in the config), pip list…
alembic 1.13.1
anyio 4.3.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.4.1
async-generator 1.10
async-lru 2.0.4
attrs 23.2.0
Babel 2.14.0
batchspawner 1.3.1.dev0
beautifulsoup4 4.12.3
bleach 6.1.0
certifi 2024.2.2
certipy 0.1.3
cffi 1.16.0
charset-normalizer 3.3.2
comm 0.2.2
cryptography 42.0.5
debugpy 1.8.1
decorator 5.1.1
defusedxml 0.7.1
executing 2.0.1
fastjsonschema 2.19.1
fqdn 1.5.1
greenlet 3.0.3
h11 0.14.0
httpcore 1.0.5
httpx 0.27.0
idna 3.7
ipykernel 6.29.4
ipython 8.24.0
ipywidgets 8.1.2
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.3
json5 0.9.25
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter_client 8.6.1
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.5
jupyter_server 2.14.0
jupyter_server_terminals 0.5.3
jupyter-telemetry 0.1.0
jupyterhub 4.1.5
jupyterlab 4.1.8
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.1
jupyterlab_widgets 3.0.10
Mako 1.3.3
MarkupSafe 2.1.5
matplotlib-inline 0.1.7
mistune 3.0.2
nbclient 0.10.0
nbconvert 7.16.4
nbformat 5.10.4
nest-asyncio 1.6.0
notebook 7.1.3
notebook_shim 0.2.4
oauthlib 3.2.2
overrides 7.7.0
packaging 24.0
pamela 1.1.0
pandocfilters 1.5.1
parso 0.8.4
pexpect 4.9.0
pip 24.0
platformdirs 4.2.1
prometheus_client 0.20.0
prompt-toolkit 3.0.43
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
pycparser 2.22
Pygments 2.17.2
pyOpenSSL 24.1.0
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
PyYAML 6.0.1
pyzmq 26.0.2
referencing 0.35.0
requests 2.31.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.18.0
ruamel.yaml 0.18.6
ruamel.yaml.clib 0.2.8
Send2Trash 1.8.3
setuptools 65.5.0
six 1.16.0
sniffio 1.3.1
soupsieve 2.5
SQLAlchemy 2.0.29
stack-data 0.6.3
terminado 0.18.1
tinycss2 1.3.0
tornado 6.4
traitlets 5.14.3
types-python-dateutil 2.9.0.20240316
typing_extensions 4.11.0
uri-template 1.3.0
urllib3 2.2.1
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.8.0
widgetsnbextension 4.0.10
wrapspawner 1.0.2.dev0
and extension list:
Config dir: /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter
jupyter_lsp enabled
- Validating jupyter_lsp…
jupyter_lsp 2.2.5 OK
jupyter_server_terminals enabled
- Validating jupyter_server_terminals…
jupyter_server_terminals 0.5.3 OK
jupyterlab enabled
- Validating jupyterlab…
jupyterlab 4.1.8 OK
notebook enabled
- Validating notebook…
notebook 7.1.3 OK
notebook_shim enabled
- Validating notebook_shim…
notebook_shim OK
Thanks,
I tried
c.Spawner.default_url = ‘/lab’
And after starging the server in jupyterhub UI (using the Profile), it complains…
404 : Not Found
You are requesting a page that does not exist!
In the jupyter hub log I see this:
Single-user server has no version header, which means it is likely < 0.8. Expected 4.1.5
Thanks,
Can you share the startup logs of the singleuser server?
When using /lab, the log shows…
[I 2024-05-27 11:38:00.212 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.5
[I 2024-05-27 11:38:00.213 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
[I 2024-05-27 11:38:00.214 ServerApp] jupyterhub | extension was successfully linked.
/usr/local/envs/python/jupyterhub/4.1.3/lib/python3.11/site-packages/jupyter_server/serverapp.py:2236: JupyterServerAuthWarning: Core endpoints without allow_unauthenticated, ws_authenticated, nor web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/jupytest/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/jupytest/oauth_callback
self.web_app = ServerWebApplication(
[I 2024-05-27 11:38:00.259 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-05-27 11:38:00.259 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-05-27 11:38:00.259 ServerApp] Serving notebooks from local directory: /home/jupytest
[I 2024-05-27 11:38:00.259 ServerApp] Jupyter Server 2.14.0 is running at:
[I 2024-05-27 11:38:00.259 ServerApp] http://c01:50373/user/jupytest/lab?token=...
[I 2024-05-27 11:38:00.259 ServerApp] http://127.0.0.1:50373/user/jupytest/lab?token=...
[I 2024-05-27 11:38:00.259 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-05-27 11:38:00.389 ServerApp] 302 GET /user/jupytest/ -> /user/jupytest/lab? (@10.20.81.105) 0.34ms
[I 2024-05-27 11:38:00.774 ServerApp] 302 GET /user/jupytest/?redirects=1 -> /user/jupytest/lab?redirects=1 (@::ffff:10.10.10.101) 0.32ms
[I 2024-05-27 11:38:00.895 ServerApp] Setting new xsrf cookie for b'42e2f2cd1c58481eafc11d4b58c6c3aa:J8A1AHGyiYMJS9s-aoPs3w2rMJjdjl7yLIJX8dgQvOw=' {'path': '/user/jupytest/', 'max_age': 3600}
[W 2024-05-27 11:38:00.906 ServerApp] 404 GET /user/jupytest/lab?redirects=1 (@::ffff:10.10.10.101) 11.37ms
Thanks
ok, it’s definitely not loading any extensions other than jupyterhub. that’s the problem, we just need to figure out why. Do you have disable_user_config set?
I searched for it in jupyterhub_config.py and only found the line commented:
# c.Spawner.disable_user_config = False
I defined a single environment for both jupyterhub and jupyterserver, I am not sure if it would help to setup an independent environment for jupyter singleuser server
Thanks,
Can you get the environment variables containing JUPYTER for the singleuser server? You can put e.g.
import os
for key in sorted(os.environ):
if 'JUPYTER' in key:
print(f"{key}={os.environ[key]}")
in ~/.jupyter/jupyter_server_config.py
I’d also like to see the output of jupyter --paths in the spawned environment, if you can get it.
If you set c.Spawner.debug = True in the JupyterHub config, single user server logs will contain the current loaded Jupyter Server config. Maybe your single user environment is picking up stale config from somewhere that is preventing the extensions to be loaded.
Thanks @mahendrapaipuri
I set
c.Spawner.debug = True
but the single user log look the same (I get it from the output of Slurm job)
Same thing happens regarding the prints that minrk suggested…
Maybe the single user server logs are somewhere else (other than the output file of slurm job)? 
Thanks @minrk
The output of jupyter --paths:
config:
/home/jupytest/.jupyter
/usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter
/usr/local/etc/jupyter
/etc/jupyter
data:
/home/jupytest/.local/share/jupyter
/usr/local/envs/python/jupyterhub/4.1.3/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/home/jupytest/.local/share/jupyter/runtime
(I added the python code to ~jupytest/.jupyter/jupyter_server_config.py , but the output of single server looks the same 
)
no, you have the singleuser server logs, that’s all the lines with ServerApp]
Can you change Spawner.cmd to: jupyterhub-singleuser --debug --show-config ? It will exit immediately, but it may show us the info we need.
I see the ServerApp lines in the log of the slurm job, but there is no new info:
[I 2024-05-30 13:16:12.912 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-05-30 13:16:12.912 ServerApp] Serving notebooks from local directory: /home/jupytest
[I 2024-05-30 13:16:12.912 ServerApp] Jupyter Server 2.14.0 is running at:
[I 2024-05-30 13:16:12.912 ServerApp] http://c01:56217/user/jupytest/lab?token=...
[I 2024-05-30 13:16:12.912 ServerApp] http://127.0.0.1:56217/user/jupytest/lab?token=...
[I 2024-05-30 13:16:12.912 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-05-30 13:16:12.990 ServerApp] 302 GET /user/jupytest/ -> /user/jupytest/lab? (@10.20.81.105) 0.34ms
[I 2024-05-30 13:16:13.404 ServerApp] 302 GET /user/jupytest/?redirects=1 -> /user/jupytest/lab?redirects=1 (@::ffff:10.10.10.101) 0.31ms
(which I find strange, it’s like it is ignoring the changes in the configuration file… though when I change ‘c.Spawner.default_url’ I do see changes from “A server is running” to “Not Found” and viceversa)
Regarding the jupyter user environment, when I configure it to run directly on the main node (skipping the slurm-related spawners, batchspawner and profilespawner) it works… maybe the way it is started by profilespawner is a bit different and causes “A server is running” (instead of actually showing the jupyterserver)
Thanks,
Could you enable debug logging on JupyterHub and post the logs here? In the debug mode, batchspawner should print the env vars set on the single user server.