JupyterHub server spawning never completes; new JupyerHub admin

JupyterHub
1

Hello, I’ve recently started a position responsible for a JupyterHub installation as part of an HPC environment; the old team left without leaving much in the way of documentation, and I’ve never worked on JupyterHub before, so please bear with me.

When launching JupyterHub, the system will stay on the page “Your server is starting up. You will be redirected automatically when it’s ready for you. Spawning server…” for as long as the value of c.Spawner.start_timeout is configured.

[root] # /hpc/project/projwork47/pyjup/sboxtest/bin/python3.1 /hpc/project/projwork47/pyjup/sboxtest/bin/jupyterhub --version
3.0.0

This is the output of starting jupyterhub from the command line, then trying to connect from the browser:

[root] # /hpc/project/projwork47/pyjup/sboxtest/bin/python3.1 /hpc/project/projwork47/pyjup/sboxtest/bin/jupyterhub --debug -f /hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_config.py
[D 2023-12-05 15:25:43.094 JupyterHub application:902] Looking for /hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_config in /hpc/project/projwork47/pyjup/sboxtest/bin
[D 2023-12-05 15:25:43.096 JupyterHub application:923] Loaded config file: /hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_config.py
[INFO] [JupyterHub] Running JupyterHub version 3.0.0
[INFO] [JupyterHub] Using Authenticator: jupyterhub.auth.PAMAuthenticator-3.0.0
[INFO] [JupyterHub] Using Spawner: systemdspawner.systemdspawner.SystemdSpawner
[INFO] [JupyterHub] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-3.0.0
[INFO] [JupyterHub] Loading cookie_secret from /hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_cookie_secret
[DEBUG] [JupyterHub] Connecting to db: sqlite:///jupyterhub.sqlite
[DEBUG] [JupyterHub] database schema version found: 651f5419b74d
[INFO] [JupyterHub] Generating new CONFIGPROXY_AUTH_TOKEN
[DEBUG] [JupyterHub] Loading roles into database
[DEBUG] [JupyterHub] Purging expired APITokens
[DEBUG] [JupyterHub] Purging expired OAuthCodes
[DEBUG] [JupyterHub] Loading role assignments from config
[DEBUG] [JupyterHub] Assigning 17 allowed_users to the user role
[DEBUG] [JupyterHub] Initializing spawners
[DEBUG] [JupyterHub] Loaded users:
[INFO] [JupyterHub] Initialized 0 spawners in 0.006 seconds
[INFO] [JupyterHub] Starting proxy @ https://0.0.0.0:443/
[DEBUG] [JupyterHub] Proxy cmd: ['configurable-http-proxy', '--ip', '0.0.0.0', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error', '--log-level', 'info', '--ssl-key', '/hpc/project/projwork47/pyjup/sboxtest/ssl/ssl.key', '--ssl-cert', '/hpc/project/projwork47/pyjup/sboxtest/ssl/ssl.crt']
[DEBUG] [JupyterHub] Writing proxy pid file: jupyterhub-proxy.pid
15:25:43.846 [ConfigProxy] info: Proxying https://0.0.0.0:443 to (no default)
15:25:43.851 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[DEBUG] [JupyterHub] Proxy started and appears to be up
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[INFO] [JupyterHub] Hub API listening on http://127.0.0.1:8081/hub/
[DEBUG] [JupyterHub] Fetching routes to check
15:25:44.408 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:25:44.410 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Checking routes
[INFO] [JupyterHub] Adding route for Hub: / => http://127.0.0.1:8081
[DEBUG] [JupyterHub] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
15:25:44.412 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
15:25:44.412 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
15:25:44.413 [ConfigProxy] info: 201 POST /api/routes/
[INFO] [JupyterHub] JupyterHub is now running at https://0.0.0.0:443/
[DEBUG] [JupyterHub] It took 1.325 seconds for the Hub to start
[DEBUG] [JupyterHub] Refreshing auth for m5573
[DEBUG] [JupyterHub] Checking access via scope servers
[DEBUG] [JupyterHub] Unrestricted access to /hub/spawn-pending/m5573 via servers
[DEBUG] [JupyterHub] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for m5573:
[DEBUG] [JupyterHub] user:m5573 Initialized spawner with unit jupyter-m5573-singleuser
[INFO] [JupyterHub] 200 GET /hub/spawn-pending/m5573 (m5573@10.144.0.27) 36.47ms
[DEBUG] [JupyterHub] 200 GET /hub/static/js/not_running.js?v=20231205152543 (@10.144.0.27) 1.79ms
[DEBUG] [JupyterHub] 200 GET /hub/static/js/utils.js?v=20231205152543 (@10.144.0.27) 1.37ms
[DEBUG] [JupyterHub] Checking access via scope servers
[DEBUG] [JupyterHub] Unrestricted access to /hub/spawn/m5573 via servers
[DEBUG] [JupyterHub] Triggering spawn with default options for m5573
[DEBUG] [JupyterHub] Initiating spawn for m5573
[DEBUG] [JupyterHub] 1/100 concurrent spawns
[DEBUG] [JupyterHub] 1/4 active servers
[INFO] [JupyterHub] Creating oauth client jupyterhub-user-m5573
[DEBUG] [JupyterHub] Calling Spawner.start for m5573
[DEBUG] [JupyterHub] user:m5573 Using port 60473 to start spawning user server
[INFO] [JupyterHub] user:m5573 Unit jupyter-m5573-singleuser in a failed state. Resetting state.
Running as unit: jupyter-m5573-singleuser.service
[INFO] [JupyterHub] 302 GET /hub/spawn/m5573 -> /hub/spawn-pending/m5573 (m5573@10.144.0.27) 1004.59ms
[DEBUG] [JupyterHub] Checking access via scope servers
[DEBUG] [JupyterHub] Unrestricted access to /hub/spawn-pending/m5573 via servers
[INFO] [JupyterHub] m5573 is pending spawn
[INFO] [JupyterHub] 200 GET /hub/spawn-pending/m5573 (m5573@10.144.0.27) 13.87ms
[DEBUG] [JupyterHub] Checking access via scope read:servers
[DEBUG] [JupyterHub] Unrestricted access to /hub/api/users/m5573/server/progress via read:servers
[WARNING] [JupyterHub] User m5573 is slow to start (timeout=10)
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:30:44.416 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Checking routes
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:35:44.416 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Checking routes
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:40:44.415 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Checking routes
[DEBUG] [JupyterHub] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
15:45:44.416 [ConfigProxy] info: 200 GET /api/routes
[DEBUG] [JupyterHub] Checking routes

This is the config file:

c = get_config()  #noqa
c.Application.log_datefmt = '%m-$d-%Y %H:%M:%S'
c.Application.log_format = '[%(levelname)s] [%(name)s] %(message)s'
c.Application.log_level = 'INFO'
c.JupyterHub.active_server_limit = 4
c.JupyterHub.active_user_window = 5
c.JupyterHub.config_file = '/hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_config.py'
c.JupyterHub.cookie_max_age_days = 2
c.JupyterHub.cookie_secret_file = '/hpc/project/projwork47/pyjup/sboxtest/etc/jupyterhub_cookie_secret'
c.JupyterHub.data_files_path = '/hpc/project/projwork47/pyjup/sboxtest/share/jupyterhub'
c.JupyterHub.init_spawners_timeout = 15
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.log_datefmt = '%m-%d-%Y %H:%M:%S'
c.JupyterHub.log_format = '[%(levelname)s] [%(name)s] %(message)s'
c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.named_server_limit_per_user = 2
c.JupyterHub.port = 443
c.JupyterHub.service_check_interval = 60
c.JupyterHub.shutdown_on_logout = True
c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
c.SystemdSpawner.dynamic_users = False
c.SystemdSpawner.mem_limit = '2G'
c.SystemdSpawner.cpu_limit = 1.0
c.JupyterHub.ssl_cert = '/hpc/project/projwork47/pyjup/sboxtest/ssl/ssl.crt'
c.JupyterHub.ssl_key = '/hpc/project/projwork47/pyjup/sboxtest/ssl/ssl.key'
c.Spawner.cmd = ['/hpc/project/projwork47/pyjup/sboxtest/bin/jupyterhub-singleuser']
c.Spawner.cpu_guarantee = 0.5
c.Spawner.cpu_limit = 1.0
c.Spawner.env_keep = ['PATH', 'PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL', 'JUPYTERHUB_SINGLEUSER_APP']
c.Spawner.http_timeout = 3600
c.Spawner.mem_guarantee = '500M'
c.Spawner.mem_limit = '2G'
c.Spawner.notebook_dir = '/hpc/project/projwork47/jupyterLabUsers'
c.Spawner.poll_interval = 30
c.Spawner.port = 0
c.Spawner.start_timeout = 3600
2

This is probably the most relevant bit:

So you want to check the logs for jupyter-m5573-singleuser, e.g.

journalctl -u  jupyter-m5573-singleuser

which will probably have a traceback pointing you to the real problem.

1 Like
3

This is the error pulled from journalctl:

Dec 13 09:43:31 skink6 bash[1448746]: [C 2023-12-13 09:43:31.172 SingleUserLabApp application:115] Bad config encountered during initialization: The 'kernel_spec_manager_class' trait of <jupyterlab.labhubapp.SingleUserLabApp object at 0x7f7dd91efc10> instance must be a type, but 'nb_conda_kernels.CondaKernelSpecManager' could not be imported

I found a suggestion for checking the serverextention list from Bad config encountered during initialization: · Issue #2875 · jupyter/notebook · GitHub but am unsure of where to go from here.

jupyter serverextension list
Config option `kernel_spec_manager_class` not recognized by `ListServerExtensionsApp`.
config dir: ...etc/jupyter
    ipyparallel  enabled
    - Validating...
      ipyparallel 8.4.1 OK
    jupyter_nbextensions_configurator  enabled
    - Validating...
      jupyter_nbextensions_configurator 0.6.1 OK
    jupyter_server_ydoc  enabled
    - Validating...
      X is jupyter_server_ydoc importable?
    jupyterlab  enabled
    - Validating...
      jupyterlab 3.6.1 OK
4

Update: While I get the “Bad config encountered during initialization” error, other users who rely on this application can now log in after a permissions correction on a parent directory.

This issue now seems to be just for my relatively new administrative account, as previously established users now have no issue spawning a server.

Not a super high priority since users are OK now, but it would still be nice for me to be able to learn why my own login is still failing; at the very least so that I understand the application more.

Thank you for your help, either way!

5

Seems like you a config entry something like c.ServerApp.kernel_spec_manager_class = "nb_conda_kernels.CondaKernelSpecManager" in single user server environment to override default kernel_spec_manager_class. But you have not installed nb_onda_kernels in that environment. I think your single user server env is /hpc/project/projwork47/pyjup/sboxtest according to your config. Can you try installing nb_conda_kernels in that environment and see if it fixes your error?

1 Like
6

nb-conda-kernels 2.3.1 shows up under pip list for sboxtest; Also, I couldn’t find any reference to those kernel configuration lines. I also checked to make sure additional configs weren’t written under any ${HOME}/.jupyter/ directory files, and the config file I referenced in my first post appears to be the only one in play system-wide.

So, I guess the question is, why is my single-user login behaving differently than other users?

7

Could you try enabling debug logs for single user server using c.Spawner.debug = True and share the entire single user logs? In debug logging, we will get all the config locations that are being read to load the configuration of single user server. That might give some insights where this config is coming from.