Slurm (Batch/Profile Spawner) - Jupyter Server is running

Is /usr/local/envs on a shared filesystem?

Make sure to run the tests of jupyter --paths, pip list, env, etc. in the actual context where the singleuser server runs (e.g. by submitting the test as a job itself).

Hi @mahendrapaipuri

Do you mean setting “c.Application.log_level=DEBUG” in jupyterhub_config.py?

Thanks

Yes, with DEBUG in quotes. c.Application.log_level = 'DEBUG'

Hi @minrk ,

Yes, /usr/local is a NFS shared filesystem (BTW, /home is Lustre)

I added the commands to c.SlurmSpawner.batch_script and now I see their outputs in the slurm job log.

JUPYTER variables:

JUPYTERHUB_ACTIVITY_URL=http://10.10.10.100:8081/hub/api/users/jupytest/activity
JUPYTERHUB_API_TOKEN=dc....
JUPYTERHUB_API_URL=http://10.10.10.100:8081/hub/api
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_CLIENT_ID=jupyterhub-user-jupytest
JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
JUPYTERHUB_DEBUG=1
JUPYTERHUB_HOST=
JUPYTERHUB_OAUTH_ACCESS_SCOPES='["access:servers!server=jupytest/", "access:servers!user=jupytest"]'
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/jupytest/oauth_callback
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES='[]'
JUPYTERHUB_OAUTH_SCOPES='["access:servers!server=jupytest/", "access:servers!user=jupytest"]'
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX=/user/jupytest/
JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/jupytest/
JUPYTERHUB_USER=jupytest

jupyter --paths

config:
    /home/jupytest/.jupyter
    /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/jupytest/.local/share/jupyter
    /usr/local/envs/python/jupyterhub/4.1.3/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/jupytest/.local/share/jupyter/runtime

pip list: same output (Slurm (Batch/Profile Spawner) - Jupyter Server is running - #7 by tranquilinho), next I recollect the jupyter-related packages:

batchspawner              1.3.1.dev0
batchspawner              1.3.1.dev0
batchspawner              1.3.1.dev0
jupyter_client            8.6.1
jupyter_core              5.7.2
jupyter-events            0.10.0
jupyter-lsp               2.2.5
jupyter_server            2.14.0
jupyter_server_terminals  0.5.3
jupyter-telemetry         0.1.0
jupyterhub                4.1.5
jupyterlab                4.1.8
jupyterlab_pygments       0.3.0
jupyterlab_server         2.27.1
jupyterlab_widgets        3.0.10
notebook                  7.1.3
notebook_shim             0.2.4
wrapspawner               1.0.2.dev0
wrapspawner               1.0.2.dev0
wrapspawner               1.0.2.dev0

It calls my attention that batchspawner and wrapspawner appear thrice each. Related to this, in slurm logs I see these messages:

DEPRECATION: Loading egg at /usr/local/envs/python/jupyterhub/4.1.3/lib/python3.11/site-packages/wrapspawner-1.0.2.dev0-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at /usr/local/envs/python/jupyterhub/4.1.3/lib/python3.11/site-packages/batchspawner-1.3.1.dev0-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330

I also included the command “jupyter server extension list” in c.SlurmSpawner.batch_script, it only prints this (in the logs):

Config dir: /home/jupytest/.jupyter

Config dir: /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter

Config dir: /usr/local/etc/jupyter

Thanks!

Thanks @mahendrapaipuri

I see extra info now in the jupyterhub logs:

    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.
    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

Spawner submitting environment: {'PATH':
'/usr/local/envs/python/jupyterhub/4.1.3/bin:/usr/local/spack/0.21.1/opt/spack/linux-rhel9-x86_64/gcc-11.4.1/node-js-18.12.1-tfmidi3pleg3yiydagmdwvetlho6o$
ck/linux-rhel9-x86_64/gcc-11.4.1/node-js-18.12.1-tfmidi3pleg3yiydagmdwvetlho6o2hb/bin:/usr/local/spack/0.21.1/opt/spack/linux-rhel9-x86_64/gcc-11
.4.1/openssl-3.1.3-aqsmuvkmwtpc3cuwi2eq4wdldde2muxv/bin:/usr/local/spack/0.21.1/bin:/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/us
r/sbin:/usr/bin', 'VIRTUAL_ENV': '/usr/local/envs/python/jupyterhub/4.1.3', 'LANG':
'en_US.UTF-8', 'JUPYTERHUB_API_TOKEN': '94...  .....62', 'JPY_API_TOKEN':
'94......................2', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-jupytest',
'JUPYTERHUB_COOKIE_ HOST_PREFIX_ENABLED': '0', 'JUPYTERHUB_HOST': '',
'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/jupytest/oauth_callback', 'JUPYTERHUB_OAUTH_SCOP
ES': '["access:servers!server=jupytest/", "access:servers!user=jupytest"]',
'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!ser ver=jupytest/",
"access:servers!user=jupytest"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]',
'JUPYTERHUB_USER': 'jupytest', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL':
'http://10.10.10.100:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://10.10.10.100:8081
/hub/api/users/jupytest/activity', 'JUPYTERHUB_BASE_URL': '/',
'JUPYTERHUB_SERVICE_PREFIX': '/user/jupytest/', 'JUPYTERHUB_SERVICE_ URL':
'http://0.0.0.0:0/user/jupytest/', 'JUPYTERHUB_DEBUG': '1', 'USER': 'jupytest', 'HOME':
'/home/jupytest ', 'SHELL': '/bin/bash'}

Great, can you add:

jupyterhub-singleuser --debug --show-config
jupyter server extension list

to your batch script so we can see the output of those two in the job context?

Hi @minrk

Sorry for the delay.

I also added those two to the batch script. First command shows…

JUPYTERHUB_ACTIVITY_URL=http://10.10.10.100:8081/hub/api/users/jupytest/activity
JUPYTERHUB_API_TOKEN=dc...4
JUPYTERHUB_API_URL=http://10.10.10.100:8081/hub/api
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_CLIENT_ID=jupyterhub-user-jupytest
JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
JUPYTERHUB_DEBUG=1
JUPYTERHUB_HOST=
JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=jupytest/", "access:servers!user=jupytest"]
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/jupytest/oauth_callback
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=jupytest/", "access:servers!user=jupytest"]
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX=/user/jupytest/
JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/jupytest/
JUPYTERHUB_USER=jupytest

[D 2024-06-07 15:04:04.858 ServerApp] Looking for jupyter_config in /etc/jupyter
[D 2024-06-07 15:04:04.858 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-06-07 15:04:04.858 ServerApp] Looking for jupyter_config in
  /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter
[D 2024-06-07 15:04:04.858 ServerApp] Looking for jupyter_config in /home/jupytest/.jupyter
[D 2024-06-07 15:04:04.859 ServerApp] Looking for jupyter_server_config in /etc/jupyter
[D 2024-06-07 15:04:04.859 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-06-07 15:04:04.859 ServerApp] Looking for jupyter_server_config in
  /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter
[D 2024-06-07 15:04:04.860 ServerApp] Looking for jupyter_server_config in /home/jupytest/.jupyter
[D 2024-06-07 15:04:04.870 ServerApp] Loaded config file:
     /home/jupytest/.jupyter/jupyter_server_config.py [D 2024-06-07
15:04:04.873 ServerApp] Paths used for configuration of jupyter_server_config:
/etc/jupyter/jupyter_server_config.json
[D 2024-06-07 15:04:04.873 ServerApp] Paths used for configuration of jupyter_server_config:
/usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-06-07 15:04:04.873 ServerApp] Paths used for configuration of jupyter_server_config:
/usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter/jupyter_server_config.json
[D 2024-06-07 15:04:04.873 ServerApp] Paths used for configuration of jupyter_server_config:
/home/jupytest/.jupyter/jupyter_server_config.json
[D 2024-06-07 15:04:04.877 ServerApp] Extension package jupyterhub took 0.0000s to import [I 2024-06-07
15:04:04.877 JupyterHubSingleUser] Starting jupyterhub single-user server extension
version 4.1.5
[W 2024-06-07 15:04:04.877 JupyterHubSingleUser] No default url found in
config or known extensions, searching other extensions for default_url [W 2024-06-07
15:04:04.877 JupyterHubSingleUser] Found no extension with a default URL, UI will likely
be unavailable
[D 2024-06-07 15:04:04.879 JupyterHubSingleUser] Config changed:
{'ServerApp': {'identity_provider_class': <class
'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access':
True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries':
0, 'answer_yes': True, 'port': 80, 'ip': '0.0.0.0', 'base_url': '/user/jupytest/',
'default_url': '/', 'keyfile': '', 'certfile': '', 'client_ca': '', 'log_level': 'DEBUG',
'show_config': True}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp':
{'extra_template_paths': <LazyConfigValue {'extend':
['/usr/local/envs/python/jupyterhub/4.1.3/lib/python3.11/site-packages/jupyterhub/singleuser/templates']}>},
'ExtensionApp': {'log_level': 'DEBUG'}

[D 2024-06-07 15:04:04.914 JupyterHubSingleUser] jupyterhub and jupyterhub-singleuser both on version 4.1.5
[I 2024-06-07 15:04:04.915 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-06-07 15:04:04.915 ServerApp] jupyterhub | extension was successfully loaded.
[W 2024-06-07 15:04:04.915 ServerApp] Permission to listen on port 80 denied.
[C 2024-06-07 15:04:04.915 ServerApp] ERROR: the Jupyter server could not be started because port 80 is not available.
[D 2024-06-07 15:04:04.915 ServerApp] Exiting application: jupyter-server

The second command (jupyter server extension list) shows…

Config dir: /home/jupytest/.jupyter

Config dir: /usr/local/envs/python/jupyterhub/4.1.3/etc/jupyter

Config dir: /usr/local/etc/jupyter

Thanks,

Well, seems like the ProfileSpawner is not properly taking c.Spawner.default_url = '/lab'. From your logs, no default_url is set and that is why you do not get UI. Maybe try setting c.SlurmSpawner.default_url = '/lab'?

Thanks @mahendrapaipuri

I added that line (and restarted jupyterhub) and the issue persists… It is interesting that the warning appears again in the logs:

[I 2024-06-10 11:38:25.540 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.5
[W 2024-06-10 11:38:25.540 JupyterHubSingleUser] No default url found in config or known extensions, searching other extensions for default_url
[W 2024-06-10 11:38:25.540 JupyterHubSingleUser] Found no extension with a default URL, UI will likely be unavailable

I am using SlurmSpawner through ProfileSpawner, maybe the default_url needs to be set in ProfileSpawner?

Thanks,

Yes, please do. Set it on ProfileSpawner and see if it works. When you set it on c.Spawner, ProfileSpawner should inherit it Not sure why it didnt though.

Thanks @mahendrapaipuri and @minrk for your help.

Finally jupyterlab appeared in the browser (the actual server running in a compute node over slurm)

Next I summarized how I found the root cause of this issue.

The fact that I made changes to the jupyterhub config and it seemed like they were ignored made me wander… maybe some config file or path was missing or wrong.

Looking again at the slurm job logs, I started checking each config path that appeared, and found that /usr/local/envs/python/jupyterhub/4.1.3/etc was actually a symlink (that’s why jupyterhub worked OK when run directly on master node, but not when running on compute node by means of Slurm / Profile spawners)

:sweat_smile: I believe I created that symlink in one of the many tests I did trying to get everything running, and after so many changes and rollbacks I forgot about it…

Now I will try to remove the debugging stuff from the config files.

Thanks again!

Best wishes,

1 Like