JupyterHub with Podman and DockerSpawner: Immediate 'JupyterLab Server Unavailable' Error on Login, Resolved by Refresh

JupyterHub
,
1

Hi guys,

I am using Jupyterhub with podman and DockerSpawner. When I log in, I immediately get an error message that the Jupyterlab server is not available:

503 : Service Unavailable
Your server appears to be down. Try restarting it from the hub

But once I refresh the page I can get access to the server with no problems. It is always the same process regardless of the user:

image
image898×182 14.6 KB

My assumption is that Jupyterhub tries to access the container server before it starts, it does not find it and I get a timeout. When I log out without shutting down the container and login again I get direct access the the container server. I tried to solve this increasing the http_timeout and http_start in the jupyter_configuration.py, but it seems like they are not being picked up. Here my configuration file:

from dockerspawner import DockerSpawner
from nativeauthenticator import NativeAuthenticator
from jupyterhub.auth import Authenticator

import os

class LDAPAuthenticator(Authenticator):

    async def authenticate(self, handler, data):
        if data['password']=='mypassword':

            return data['username']


c.JupyterHub.authenticator_class = LDAPAuthenticator


#c.GenericOAuthenticator.enable_auth_state = True

c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.hub_ip = '192.168.1.10'





c.JupyterHub.spawner_class = DockerSpawner

c.DockerSpawner.network_name = 'data_process_excel'

c.DockerSpawner.remove = True

c.Spawner.http_timeout = 300
c.Spawner.start_timeout = 600

notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir

c.DockerSpawner.volumes = { '/data/JupyterHub/user/jupyterhub-user-{username}': notebook_dir }
c.DockerSpawner.image = "jupyter/datascience-notebook:latest"

c.Spawner.cpu_limit = 1
c.Spawner.mem_limit = '8G'

# Persistence
c.JupyterHub.db_url = "sqlite:///data/jupyterhub.sqlite"

# Enable user registration
c.Authenticator.allowed_users = set()
c.Authenticator.admin_users = {'Admin'}
#c.NativeAuthenticator.open_signup = True

#c.NativeAuthenticator.create_system_users = True

#c.DockerSpawner.use_docker_client = True

And I run the Jupyterhub with the following podman commands, after building the image:

podman run -d --name jupyterhub \
  --network data_process_excel \
  --ip 192.168.1.10 \
  -v /home/rootx/jupyterhub/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py \
  -v /run/podman/podman.sock:/var/run/docker.sock \
  -v /data/JupyterHub/data:/srv/jupyterhub/data \
  -p 8000:8000 \
  jupyterhub

Everything, the podman commands and configuration seem to work, except the fact that Jupyterhub throws the “unavailable error” before allowing access to the container server. Can you please help?

Thanks

2

Can you turn on debug logging in JupyterHub, and show us the hub and singleuser server logs?

Have you tried enabling the debug log-level in the Podman service?

3

Hallo Simon,

thank you so much for your response.

Here is the Logs of Jupyterhub after starting and spawning the singleuser Server:

[I 2024-01-13 13:10:33.799 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2024-01-13 13:10:33.799 JupyterHub app:2889] Using Authenticator: builtins.LDAPAuthenticator
[I 2024-01-13 13:10:33.800 JupyterHub app:2889] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-13.0.0
[I 2024-01-13 13:10:33.800 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[D 2024-01-13 13:10:33.814 JupyterHub app:1704] Generating new cookie_secret
[I 2024-01-13 13:10:33.814 JupyterHub app:1709] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
[D 2024-01-13 13:10:33.815 JupyterHub app:1833] Connecting to db: sqlite:///data/jupyterhub.sqlite
[D 2024-01-13 13:10:33.915 JupyterHub orm:1016] database schema version found: 0eee8c825d24
[I 2024-01-13 13:10:34.226 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2024-01-13 13:10:34.227 JupyterHub app:2104] Loading roles into database
[I 2024-01-13 13:10:34.388 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[D 2024-01-13 13:10:34.418 JupyterHub app:2343] Purging expired APITokens
[D 2024-01-13 13:10:34.475 JupyterHub app:2343] Purging expired OAuthCodes
[D 2024-01-13 13:10:34.500 JupyterHub app:2179] Loading role assignments from config
[D 2024-01-13 13:10:34.591 JupyterHub app:2502] Initializing spawners
[D 2024-01-13 13:10:34.594 JupyterHub app:2633] Loaded users:

[I 2024-01-13 13:10:34.594 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2024-01-13 13:10:34.601 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-01-13 13:10:34.602 JupyterHub metrics:278] Found 11 active users in the last ActiveUserPeriods.seven_days
[I 2024-01-13 13:10:34.604 JupyterHub metrics:278] Found 11 active users in the last ActiveUserPeriods.thirty_days
[W 2024-01-13 13:10:34.604 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2024-01-13 13:10:34.604 JupyterHub proxy:750] Starting proxy @ http://192.168.1.10:8000/
[D 2024-01-13 13:10:34.604 JupyterHub proxy:751] Proxy cmd: ['configurable-http-proxy', '--ip', '192.168.1.10', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://192.168.1.10:8081/hub/error', '--log-level', 'info']
[D 2024-01-13 13:10:34.605 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
13:10:34.831 [ConfigProxy] info: Proxying http://192.168.1.10:8000 to (no default)
13:10:34.834 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-01-13 13:10:34.929 JupyterHub proxy:787] Proxy started and appears to be up
[D 2024-01-13 13:10:34.930 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2024-01-13 13:10:34.939 JupyterHub app:3178] Hub API listening on http://192.168.1.10:8081/hub/
13:10:34.940 [ConfigProxy] info: 200 GET /api/routes
[D 2024-01-13 13:10:34.940 JupyterHub proxy:389] Fetching routes to check
[D 2024-01-13 13:10:34.940 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
13:10:34.941 [ConfigProxy] info: 200 GET /api/routes
[D 2024-01-13 13:10:34.941 JupyterHub proxy:392] Checking routes
[I 2024-01-13 13:10:34.941 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.1.10:8081
[D 2024-01-13 13:10:34.941 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
13:10:34.943 [ConfigProxy] info: Adding route / -> http://192.168.1.10:8081
13:10:34.943 [ConfigProxy] info: Route added / -> http://192.168.1.10:8081
13:10:34.944 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-01-13 13:10:34.944 JupyterHub app:3245] JupyterHub is now running at http://192.168.1.10:8000/
[D 2024-01-13 13:10:34.944 JupyterHub app:2852] It took 1.150 seconds for the Hub to start
[I 2024-01-13 13:10:52.603 JupyterHub log:191] 302 GET / -> /hub/ (@10.69.150.160) 1.37ms
[W 2024-01-13 13:10:52.612 JupyterHub base:415] Invalid or expired cookie token
[I 2024-01-13 13:10:52.613 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@10.69.150.160) 1.71ms
[I 2024-01-13 13:10:52.872 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@10.69.150.160) 29.94ms
[D 2024-01-13 13:11:37.291 JupyterHub roles:281] Assigning default role to User yamr
[D 2024-01-13 13:11:37.296 JupyterHub base:591] Setting cookie for yamr: jupyterhub-hub-login
[D 2024-01-13 13:11:37.296 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-01-13 13:11:37.296 JupyterHub base:837] User logged in: yamr
[I 2024-01-13 13:11:37.296 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (yamr@10.69.150.160) 18.58ms
[D 2024-01-13 13:11:37.314 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for yamr:
[I 2024-01-13 13:11:37.315 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (yamr@10.69.150.160) 11.71ms
[D 2024-01-13 13:11:37.327 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2024-01-13 13:11:37.327 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2024-01-13 13:11:37.327 JupyterHub pages:217] Triggering spawn with default options for yamr
[D 2024-01-13 13:11:37.327 JupyterHub base:961] Initiating spawn for yamr
[D 2024-01-13 13:11:37.327 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-01-13 13:11:37.327 JupyterHub base:970] 0 active servers
[I 2024-01-13 13:11:37.350 JupyterHub provider:659] Creating oauth client jupyterhub-user-yamr
[D 2024-01-13 13:11:37.366 JupyterHub user:794] Calling Spawner.start for yamr
[D 2024-01-13 13:11:37.514 JupyterHub dockerspawner:1027] Getting container 'jupyter-yamr'
[I 2024-01-13 13:11:37.518 JupyterHub dockerspawner:1033] Container 'jupyter-yamr' is gone
[D 2024-01-13 13:11:37.548 JupyterHub dockerspawner:1205] Starting host with config: {'auto_remove': True, 'binds': {'/data/JupyterHub/user/jupyterhub-user-yamr': {'bind': '/home/jovyan/work', 'mode': 'rw'}}, 'links': {}, 'mounts': [], 'mem_limit': 8589934592, 'cpu_period': 100000, 'cpu_quota': 100000, 'network_mode': 'data_process_excel'}
[I 2024-01-13 13:11:37.594 JupyterHub dockerspawner:1311] Created container jupyter-yamr (id: b906634) from image jupyter/datascience-notebook:latest
[I 2024-01-13 13:11:37.594 JupyterHub dockerspawner:1335] Starting container jupyter-yamr (id: b906634)
[D 2024-01-13 13:11:37.984 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-01-13 13:11:37.984 JupyterHub dockerspawner:972] Persisting state for yamr: container name=jupyter-yamr, id=b906634b6361feb52c01d7a45bbed49fd7a91a100414fa6d7171f7b20ab40f26
[D 2024-01-13 13:11:38.002 JupyterHub utils:278] Server at http://192.168.1.11:8888/user/yamr/ responded with 407
[W 2024-01-13 13:11:38.002 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
[I 2024-01-13 13:11:38.002 JupyterHub base:990] User yamr took 0.675 seconds to start
[I 2024-01-13 13:11:38.002 JupyterHub proxy:330] Adding user yamr to proxy /user/yamr/ => http://192.168.1.11:8888
[D 2024-01-13 13:11:38.002 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/yamr
13:11:38.004 [ConfigProxy] info: Adding route /user/yamr -> http://192.168.1.11:8888
13:11:38.004 [ConfigProxy] info: Route added /user/yamr -> http://192.168.1.11:8888
13:11:38.005 [ConfigProxy] info: 201 POST /api/routes/user/yamr
[I 2024-01-13 13:11:38.005 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/yamr (yamr@10.69.150.160) 680.53ms
[D 2024-01-13 13:11:38.015 JupyterHub scopes:877] Checking access to /hub/spawn-pending/yamr via scope servers
[D 2024-01-13 13:11:38.015 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/yamr via servers
[I 2024-01-13 13:11:38.016 JupyterHub log:191] 302 GET /hub/spawn-pending/yamr -> /user/yamr/ (yamr@10.69.150.160) 3.52ms
13:11:38.025 [ConfigProxy] error: 503 GET /user/yamr/ connect ECONNREFUSED 192.168.1.11:8888
[D 2024-01-13 13:11:38.029 JupyterHub pages:584] No template for 503
[I 2024-01-13 13:11:38.033 JupyterHub log:191] 200 GET /hub/error/503?url=%2Fuser%2Fyamr%2F (@192.168.1.10) 5.39ms
[I 2024-01-13 13:11:39.830 JupyterHub log:191] 200 GET /hub/api (@192.168.1.11) 1.83ms
[D 2024-01-13 13:11:39.842 JupyterHub base:299] Recording first activity for <APIToken('4af8...', user='yamr', client_id='jupyterhub')>
[D 2024-01-13 13:11:39.849 JupyterHub scopes:877] Checking access to /hub/api/users/yamr/activity via scope users:activity
[D 2024-01-13 13:11:39.850 JupyterHub scopes:690] Argument-based access to /hub/api/users/yamr/activity via users:activity
[D 2024-01-13 13:11:39.851 JupyterHub users:874] Activity for user yamr: 2024-01-13T13:11:39.807477Z
[D 2024-01-13 13:11:39.851 JupyterHub users:892] Activity on server yamr/: 2024-01-13T13:11:39.807477Z
[I 2024-01-13 13:11:39.857 JupyterHub log:191] 200 POST /hub/api/users/yamr/activity (yamr@192.168.1.11) 21.52ms

There an error in logs and that is when JUpyterhub throws the “Unavailable error” in the web interface.

The singleuser server does not show any errors as far as I can tell.Here are the logs of the singleuser:

WARNING: using start-singleuser.sh instead of start-notebook.sh to start a server associated with JupyterHub.
Entered start.sh with args: jupyterhub-singleuser --ip=0.0.0.0
Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2024-01-13 13:11:39.336 SingleUserLabApp mixins:609] Starting jupyterhub single-user server version 3.1.0
[I 2024-01-13 13:11:39.337 SingleUserLabApp mixins:623] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.5.2
[I 2024-01-13 13:11:39.337 SingleUserLabApp mixins:623] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.1.0
[D 2024-01-13 13:11:39.345 SingleUserLabApp application:190] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2024-01-13 13:11:39.346 SingleUserLabApp application:900] Looking for jupyter_config in /etc/jupyter
[D 2024-01-13 13:11:39.346 SingleUserLabApp application:900] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-01-13 13:11:39.346 SingleUserLabApp application:900] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2024-01-13 13:11:39.346 SingleUserLabApp application:900] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2024-01-13 13:11:39.346 SingleUserLabApp application:900] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2024-01-13 13:11:39.347 SingleUserLabApp application:900] Looking for jupyter_server_config in /etc/jupyter
[D 2024-01-13 13:11:39.347 SingleUserLabApp application:921] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2024-01-13 13:11:39.347 SingleUserLabApp application:900] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-01-13 13:11:39.347 SingleUserLabApp application:900] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2024-01-13 13:11:39.347 SingleUserLabApp application:900] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2024-01-13 13:11:39.348 SingleUserLabApp application:900] Looking for jupyter_server_config in /home/jovyan/.jupyter
[W 2024-01-13 13:11:39.348 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2024-01-13 13:11:39.349 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /etc/jupyter/jupyter_server_config.json
[D 2024-01-13 13:11:39.349 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-01-13 13:11:39.349 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
        /opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
        /opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
        /opt/conda/etc/jupyter/jupyter_server_config.json
[D 2024-01-13 13:11:39.350 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2024-01-13 13:11:39.350 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /home/jovyan/.jupyter/jupyter_server_config.json
[W 2024-01-13 13:11:39.363 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2024-01-13 13:11:39.363 SingleUserLabApp application:457] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'SingleUserLabApp': {'default_url': '/lab', 'ip': '0.0.0.0'}, 'ServerApp': {'ip': '0.0.0.0', 'port': 8888, 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook_shim': True}>}, 'InlineBackend': {'figure_formats': {'svg', 'png', 'pdf', 'jpeg'}}}
[W 2024-01-13 13:11:39.364 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2024-01-13 13:11:39.364 SingleUserLabApp manager:350] jupyter_server_terminals | extension was successfully linked.
[W 2024-01-13 13:11:39.369 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2024-01-13 13:11:39.369 SingleUserLabApp manager:350] jupyterlab | extension was successfully linked.
[W 2024-01-13 13:11:39.372 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-01-13 13:11:39.372 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-01-13 13:11:39.372 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-01-13 13:11:39.374 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2024-01-13 13:11:39.375 SingleUserLabApp manager:350] nbclassic | extension was successfully linked.
[D 2024-01-13 13:11:39.575 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[D 2024-01-13 13:11:39.575 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2024-01-13 13:11:39.575 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2024-01-13 13:11:39.575 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2024-01-13 13:11:39.576 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
[D 2024-01-13 13:11:39.576 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2024-01-13 13:11:39.576 SingleUserLabApp manager:350] notebook_shim | extension was successfully linked.
[W 2024-01-13 13:11:39.801 SingleUserLabApp serverapp:1900] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2024-01-13 13:11:39.817 SingleUserLabApp manager:369] notebook_shim | extension was successfully loaded.
[I 2024-01-13 13:11:39.818 SingleUserLabApp manager:369] jupyter_server_terminals | extension was successfully loaded.
[I 2024-01-13 13:11:39.818 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab
[I 2024-01-13 13:11:39.819 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2024-01-13 13:11:39.821 SingleUserLabApp manager:369] jupyterlab | extension was successfully loaded.
[I 2024-01-13 13:11:39.824 SingleUserLabApp manager:369] nbclassic | extension was successfully loaded.
[I 2024-01-13 13:11:39.825 SingleUserLabApp mixins:670] Starting jupyterhub-singleuser server version 3.1.0
[W 2024-01-13 13:11:39.830 SingleUserLabApp _version:68] jupyterhub version 4.0.2 != jupyterhub-singleuser version 3.1.0. This could cause failure to authenticate and result in redirect loops!
[I 2024-01-13 13:11:39.830 SingleUserLabApp serverapp:2795] Serving notebooks from local directory: /home/jovyan/work
[I 2024-01-13 13:11:39.830 SingleUserLabApp serverapp:2795] Jupyter Server 2.1.0 is running at:
[I 2024-01-13 13:11:39.830 SingleUserLabApp serverapp:2795] http://b906634b6361:8888/user/yamr/lab?token=4713a2e3f0a29f10e4754c02f2d4a156d215fb8b153f6f12
[I 2024-01-13 13:11:39.830 SingleUserLabApp serverapp:2795]  or http://127.0.0.1:8888/user/yamr/lab?token=4713a2e3f0a29f10e4754c02f2d4a156d215fb8b153f6f12
[I 2024-01-13 13:11:39.830 SingleUserLabApp serverapp:2796] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-01-13 13:11:39.833 SingleUserLabApp serverapp:2859]
4

This minimal config is working for me:

# jupyterhub_config_podman.py

c.JupyterHub.spawner_class = "docker"
c.JupyterHub.authenticator_class = "dummy"
c.Authenticator.admin_users = {'demo'}

c.DockerSpawner.image = "quay.io/jupyter/base-notebook:latest"

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
c.JupyterHub.hub_ip = s.getsockname()[0]
s.close()

c.ConfigurableHTTPProxy.debug = True

Podman command:
podman system service -t0 --log-level=debug

JupyterHub command:
DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock jupyterhub --config jupyterhub_config_podman.py --debug

Do you have the same problem with this config?

5

You are using JupyterHub 4 but single user server is using JupyterHub 3. Maybe this is not the issue. Just to eliminate the variables can you try using the single user server image that supports JupyterHub 4?

1 Like
6

Hello Simon,

thanks for your support. I need to run the JupyterHub on a container as well.

So, I ran your code in a container. Now when I log in, Jupyterhub does actually wait for the singleserver to start but eventually fails to get a response (But it does run the singleserver container - I can see it).

JupyterHub_
JupyterHub_1226×363 16.9 KB

The reason is bacause the two conatiners must communicate in the same podman network. So given that I run JupyterHub in a specific network, I added only one line to your code:

c.DockerSpawner.network_name = 'data_process_excel'

unfortunately, now I get a whole different error. For some reason the network is not recognized! ( Exception: Unknown docker network ‘data_process_excel’. Did you create it with docker network create <name>?). But the network 100% exists, because Jupyterhub is already running with this network! (In my previous _config.py file I don’t get this error)

[I 2024-01-20 23:29:00.522 JupyterHub proxy:750] Starting proxy @ http://:8000
[D 2024-01-20 23:29:00.522 JupyterHub proxy:751] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://192.168.1.11:8081/hub/error', '--log-level', 'debug']
[D 2024-01-20 23:29:00.523 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
23:29:00.754 [ConfigProxy] info: Proxying http://*:8000 to (no default)
23:29:00.756 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-01-20 23:29:00.915 JupyterHub proxy:787] Proxy started and appears to be up
[D 2024-01-20 23:29:00.917 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2024-01-20 23:29:00.931 JupyterHub app:3178] Hub API listening on http://192.168.1.11:8081/hub/
[D 2024-01-20 23:29:00.931 JupyterHub proxy:389] Fetching routes to check
[D 2024-01-20 23:29:00.931 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
23:29:00.932 [ConfigProxy] info: 200 GET /api/routes
23:29:00.933 [ConfigProxy] info: 200 GET /api/routes
[D 2024-01-20 23:29:00.934 JupyterHub proxy:392] Checking routes
[I 2024-01-20 23:29:00.934 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.1.11:8081
[D 2024-01-20 23:29:00.935 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
23:29:00.937 [ConfigProxy] info: Adding route / -> http://192.168.1.11:8081
23:29:00.937 [ConfigProxy] info: Route added / -> http://192.168.1.11:8081
23:29:00.938 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-01-20 23:29:00.938 JupyterHub app:3245] JupyterHub is now running at http://:8000
[D 2024-01-20 23:29:00.940 JupyterHub app:2852] It took 0.688 seconds for the Hub to start
23:31:35.013 [ConfigProxy] debug: PROXY WEB / to http://192.168.1.11:8081
[I 2024-01-20 23:31:35.025 JupyterHub log:191] 302 GET / -> /hub/ (@::ffff:10.69.150.160) 2.30ms
23:31:35.026 [ConfigProxy] debug: Not recording activity for status 302 on /
23:31:35.032 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.11:8081
[W 2024-01-20 23:31:35.033 JupyterHub base:415] Invalid or expired cookie token
[I 2024-01-20 23:31:35.034 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:10.69.150.160) 1.21ms
23:31:35.034 [ConfigProxy] debug: Not recording activity for status 302 on /
23:31:35.274 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.11:8081
[I 2024-01-20 23:31:35.302 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:10.69.150.160) 26.17ms
23:32:17.518 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.11:8081
[D 2024-01-20 23:32:17.524 JupyterHub roles:281] Assigning default role to User dummy
[I 2024-01-20 23:32:17.528 JupyterHub roles:238] Adding role user for User: dummy
[D 2024-01-20 23:32:17.542 JupyterHub roles:281] Assigning default role to User dummy
[D 2024-01-20 23:32:17.547 JupyterHub base:591] Setting cookie for dummy: jupyterhub-hub-login
[D 2024-01-20 23:32:17.547 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-01-20 23:32:17.547 JupyterHub base:837] User logged in: dummy
[I 2024-01-20 23:32:17.548 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (dummy@::ffff:10.69.150.160) 28.04ms
23:32:17.549 [ConfigProxy] debug: Not recording activity for status 302 on /
23:32:17.555 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.11:8081
[D 2024-01-20 23:32:17.559 JupyterHub base:299] Recording first activity for <User(dummy 0/1 running)>
[D 2024-01-20 23:32:17.565 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for dummy:
[I 2024-01-20 23:32:17.566 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (dummy@::ffff:10.69.150.160) 9.27ms
23:32:17.567 [ConfigProxy] debug: Not recording activity for status 302 on /
23:32:17.572 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://192.168.1.11:8081
[D 2024-01-20 23:32:17.574 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2024-01-20 23:32:17.574 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2024-01-20 23:32:17.574 JupyterHub pages:217] Triggering spawn with default options for dummy
[D 2024-01-20 23:32:17.575 JupyterHub base:961] Initiating spawn for dummy
[D 2024-01-20 23:32:17.575 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-01-20 23:32:17.575 JupyterHub base:970] 0 active servers
[I 2024-01-20 23:32:17.592 JupyterHub provider:659] Creating oauth client jupyterhub-user-dummy
[D 2024-01-20 23:32:17.605 JupyterHub user:794] Calling Spawner.start for dummy
[D 2024-01-20 23:32:17.741 JupyterHub dockerspawner:1027] Getting container 'jupyter-dummy'
[I 2024-01-20 23:32:17.746 JupyterHub dockerspawner:1320] Found existing container jupyter-dummy (id: 448f46a)
[I 2024-01-20 23:32:17.746 JupyterHub dockerspawner:1335] Starting container jupyter-dummy (id: 448f46a)
[E 2024-01-20 23:32:18.230 JupyterHub user:884] Unhandled error starting dummy's server: Unknown docker network 'data_process_excel'. Did you create it with `docker network create <name>`?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1348, in start
        ip, port = await self.get_ip_and_port()
      File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1384, in get_ip_and_port
        ip = self.get_network_ip(network_settings)
      File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1406, in get_network_ip
        raise Exception(
    Exception: Unknown docker network 'data_process_excel'. Did you create it with `docker network create <name>`?

What is also worth mentioning is the fact that I work behind a corporate proxy. The network “data_process_excel” run with a specifc --subnet 192.168… and these IPs are excluded in the non_proxy variable in Redhat (another application with four conatiners that communicate with one another works perfectely fine with this setup).

I hope you can further help.

7

Hello Mahendra,

thank you so much for your observation. I’ll make sure, I run the singleserver with the same version as the hub.

8

Hi Simon,

you can ignore the “Unknown docker network ‘data_process_excel’” error. I was spwaning an old/ exisitng container, with old logs.

But now after adding only the network to you configuration, I get the initial situation: the “unavailable server” error, resolved by refreshing!

Here are the logs with your code:

JupyterHub ( the error and after refreshing)

[D 2024-01-22 21:02:00.716 JupyterHub proxy:751] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://192.168.1.10:8081/hub/error', '--log-level', 'debug']
[D 2024-01-22 21:02:00.717 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
21:02:00.954 [ConfigProxy] info: Proxying http://*:8000 to (no default)
21:02:00.956 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-01-22 21:02:01.202 JupyterHub proxy:787] Proxy started and appears to be up
[D 2024-01-22 21:02:01.203 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:02:01.212 [ConfigProxy] info: 200 GET /api/routes
[I 2024-01-22 21:02:01.213 JupyterHub app:3178] Hub API listening on http://192.168.1.10:8081/hub/
[D 2024-01-22 21:02:01.214 JupyterHub proxy:389] Fetching routes to check
[D 2024-01-22 21:02:01.214 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:02:01.215 [ConfigProxy] info: 200 GET /api/routes
[D 2024-01-22 21:02:01.215 JupyterHub proxy:392] Checking routes
[I 2024-01-22 21:02:01.215 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.1.10:8081
[D 2024-01-22 21:02:01.215 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
21:02:01.217 [ConfigProxy] info: Adding route / -> http://192.168.1.10:8081
21:02:01.217 [ConfigProxy] info: Route added / -> http://192.168.1.10:8081
21:02:01.218 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-01-22 21:02:01.218 JupyterHub app:3245] JupyterHub is now running at http://:8000
[D 2024-01-22 21:02:01.219 JupyterHub app:2852] It took 0.796 seconds for the Hub to start
21:02:15.767 [ConfigProxy] debug: PROXY WEB / to http://192.168.1.10:8081
[I 2024-01-22 21:02:15.778 JupyterHub log:191] 302 GET / -> /hub/ (@::ffff:10.69.150.160) 2.48ms
21:02:15.778 [ConfigProxy] debug: Not recording activity for status 302 on /
21:02:15.784 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.10:8081
[W 2024-01-22 21:02:15.787 JupyterHub base:415] Invalid or expired cookie token
21:02:15.789 [ConfigProxy] debug: Not recording activity for status 302 on /
[I 2024-01-22 21:02:15.789 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:10.69.150.160) 3.02ms
21:02:15.793 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.10:8081
[I 2024-01-22 21:02:15.827 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:10.69.150.160) 31.70ms
21:02:21.554 [ConfigProxy] debug: PROXY WEB /hub/login to http://192.168.1.10:8081
[D 2024-01-22 21:02:21.564 JupyterHub roles:281] Assigning default role to User yassine
[I 2024-01-22 21:02:21.567 JupyterHub roles:238] Adding role user for User: yassine
[D 2024-01-22 21:02:21.583 JupyterHub roles:281] Assigning default role to User yassine
[D 2024-01-22 21:02:21.587 JupyterHub base:591] Setting cookie for yassine: jupyterhub-hub-login
[D 2024-01-22 21:02:21.587 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2024-01-22 21:02:21.587 JupyterHub base:837] User logged in: yassine
[I 2024-01-22 21:02:21.588 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (yassine@::ffff:10.69.150.160) 31.06ms
21:02:21.588 [ConfigProxy] debug: Not recording activity for status 302 on /
21:02:21.593 [ConfigProxy] debug: PROXY WEB /hub/ to http://192.168.1.10:8081
[D 2024-01-22 21:02:21.597 JupyterHub base:299] Recording first activity for <User(yassine 0/1 running)>
[D 2024-01-22 21:02:21.604 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for yassine:
[I 2024-01-22 21:02:21.606 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (yassine@::ffff:10.69.150.160) 10.97ms
21:02:21.606 [ConfigProxy] debug: Not recording activity for status 302 on /
21:02:21.614 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://192.168.1.10:8081
[D 2024-01-22 21:02:21.619 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
[D 2024-01-22 21:02:21.619 JupyterHub scopes:690] Argument-based access to /hub/spawn via servers
[D 2024-01-22 21:02:21.620 JupyterHub pages:217] Triggering spawn with default options for yassine
[D 2024-01-22 21:02:21.620 JupyterHub base:961] Initiating spawn for yassine
[D 2024-01-22 21:02:21.621 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-01-22 21:02:21.621 JupyterHub base:970] 0 active servers
[I 2024-01-22 21:02:21.640 JupyterHub provider:659] Creating oauth client jupyterhub-user-yassine
[D 2024-01-22 21:02:21.654 JupyterHub user:794] Calling Spawner.start for yassine
[D 2024-01-22 21:02:21.789 JupyterHub dockerspawner:1027] Getting container 'jupyter-yassine'
[I 2024-01-22 21:02:21.792 JupyterHub dockerspawner:1033] Container 'jupyter-yassine' is gone
[D 2024-01-22 21:02:21.815 JupyterHub dockerspawner:1205] Starting host with config: {'auto_remove': False, 'binds': {}, 'links': {}, 'mounts': [], 'mem_limit': 0, 'cpu_period': 100000, 'cpu_quota': 0, 'network_mode': 'data_process_excel'}
[I 2024-01-22 21:02:21.857 JupyterHub dockerspawner:1311] Created container jupyter-yassine (id: 573776c) from image jupyter/base-notebook:latest
[I 2024-01-22 21:02:21.858 JupyterHub dockerspawner:1335] Starting container jupyter-yassine (id: 573776c)
[D 2024-01-22 21:02:22.283 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-01-22 21:02:22.283 JupyterHub dockerspawner:972] Persisting state for yassine: container name=jupyter-yassine, id=573776c4caf6fab2f3ddf3b725b6fbea668a9e3273dbf46ab745cbb24a107bc4
[D 2024-01-22 21:02:22.300 JupyterHub utils:278] Server at http://192.168.1.11:8888/user/yassine/ responded with 407
[W 2024-01-22 21:02:22.300 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
[I 2024-01-22 21:02:22.300 JupyterHub base:990] User yassine took 0.680 seconds to start
[I 2024-01-22 21:02:22.301 JupyterHub proxy:330] Adding user yassine to proxy /user/yassine/ => http://192.168.1.11:8888
[D 2024-01-22 21:02:22.301 JupyterHub proxy:880] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/yassine
21:02:22.303 [ConfigProxy] info: Adding route /user/yassine -> http://192.168.1.11:8888
21:02:22.304 [ConfigProxy] info: Route added /user/yassine -> http://192.168.1.11:8888
21:02:22.305 [ConfigProxy] info: 201 POST /api/routes/user/yassine
[I 2024-01-22 21:02:22.307 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/yassine (yassine@::ffff:10.69.150.160) 691.13ms
21:02:22.308 [ConfigProxy] debug: Not recording activity for status 302 on /
21:02:22.313 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/yassine to http://192.168.1.10:8081
[D 2024-01-22 21:02:22.316 JupyterHub scopes:877] Checking access to /hub/spawn-pending/yassine via scope servers
[D 2024-01-22 21:02:22.316 JupyterHub scopes:690] Argument-based access to /hub/spawn-pending/yassine via servers
[I 2024-01-22 21:02:22.317 JupyterHub log:191] 302 GET /hub/spawn-pending/yassine -> /user/yassine/ (yassine@::ffff:10.69.150.160) 2.39ms
21:02:22.317 [ConfigProxy] debug: Not recording activity for status 302 on /
21:02:22.321 [ConfigProxy] debug: PROXY WEB /user/yassine/ to http://192.168.1.11:8888
21:02:22.323 [ConfigProxy] error: 503 GET /user/yassine/ connect ECONNREFUSED 192.168.1.11:8888
21:02:22.324 [ConfigProxy] debug: Requesting custom error page: http://192.168.1.10:8081/hub/error/503?url=%2Fuser%2Fyassine%2F
[D 2024-01-22 21:02:22.325 JupyterHub pages:584] No template for 503
[I 2024-01-22 21:02:22.330 JupyterHub log:191] 200 GET /hub/error/503?url=%2Fuser%2Fyassine%2F (@192.168.1.10) 5.11ms
21:02:22.330 [ConfigProxy] debug: Not recording activity for status 503 on /user/yassine
[I 2024-01-22 21:02:24.008 JupyterHub log:191] 200 GET /hub/api (@192.168.1.11) 0.71ms
[D 2024-01-22 21:02:24.044 JupyterHub base:299] Recording first activity for <APIToken('613f...', user='yassine', client_id='jupyterhub')>
[D 2024-01-22 21:02:24.051 JupyterHub scopes:877] Checking access to /hub/api/users/yassine/activity via scope users:activity
[D 2024-01-22 21:02:24.051 JupyterHub scopes:690] Argument-based access to /hub/api/users/yassine/activity via users:activity
[D 2024-01-22 21:02:24.052 JupyterHub users:874] Activity for user yassine: 2024-01-22T21:02:23.989316Z
[D 2024-01-22 21:02:24.052 JupyterHub users:892] Activity on server yassine/: 2024-01-22T21:02:23.989316Z
[I 2024-01-22 21:02:24.058 JupyterHub log:191] 200 POST /hub/api/users/yassine/activity (yassine@192.168.1.11) 21.36ms
21:02:24.816 [ConfigProxy] debug: PROXY WEB /hub/home to http://192.168.1.10:8081
[D 2024-01-22 21:02:24.820 JupyterHub dockerspawner:1027] Getting container 'jupyter-yassine'
[D 2024-01-22 21:02:24.826 JupyterHub dockerspawner:1012] Container 573776c status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'Health': {'FailingStreak': 0,
                'Log': [{'End': '2024-01-22T22:02:22.591887658+01:00',
                         'ExitCode': 1,
                         'Output': 'Traceback (most recent call last):\n'
                                   '  File "/etc/jupyter/docker_healthcheck.py", '
                                   'line 14, in <module>\n'
                                   '    json_file = '
                                   'next(runtime_dir.glob("*server-*.json"))\n'
                                   '                '
                                   '^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
                                   'StopIteration',
                         'Start': '2024-01-22T22:02:22.339256304+01:00'}],
                'Status': 'starting'},
     'OOMKilled': False,
     'Paused': False,
     'Pid': 1694465,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2024-01-22T21:02:22.254333027Z',
     'Status': 'running'}

Singelserver:

Entered start.sh with args: jupyterhub-singleuser --ip=0.0.0.0
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/start-notebook.d
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/before-notebook.d
Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2024-01-22 21:02:23.155 ServerApp] Package jupyterhub took 0.0000s to import
[I 2024-01-22 21:02:23.166 ServerApp] Package jupyter_lsp took 0.0109s to import
[W 2024-01-22 21:02:23.166 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-01-22 21:02:23.171 ServerApp] Package jupyter_server_terminals took 0.0049s to import
[I 2024-01-22 21:02:23.230 ServerApp] Package jupyterlab took 0.0587s to import
[I 2024-01-22 21:02:23.759 ServerApp] Package nbclassic took 0.0022s to import
[W 2024-01-22 21:02:23.761 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-01-22 21:02:23.761 ServerApp] Package notebook took 0.0000s to import
[I 2024-01-22 21:02:23.763 ServerApp] Package notebook_shim took 0.0000s to import
[W 2024-01-22 21:02:23.763 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-01-22 21:02:23.763 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-01-22 21:02:23.768 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-01-22 21:02:23.768 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
[I 2024-01-22 21:02:23.769 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2024-01-22 21:02:23.771 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-01-22 21:02:23.773 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-01-22 21:02:23.776 ServerApp] jupyterlab | extension was successfully linked.
[W 2024-01-22 21:02:23.777 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-01-22 21:02:23.779 ServerApp] nbclassic | extension was successfully linked.
[W 2024-01-22 21:02:23.780 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2024-01-22 21:02:23.783 ServerApp] notebook | extension was successfully linked.
[I 2024-01-22 21:02:23.784 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2024-01-22 21:02:23.981 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-01-22 21:02:23.997 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-01-22 21:02:23.999 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-01-22 21:02:24.000 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-01-22 21:02:24.011 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-01-22 21:02:24.012 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-01-22 21:02:24.017 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2024-01-22 21:02:24.017 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2024-01-22 21:02:24.018 LabApp] Extension Manager is 'pypi'.
[I 2024-01-22 21:02:24.020 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-01-22 21:02:24.027 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-01-22 21:02:24.031 ServerApp] notebook | extension was successfully loaded.
[I 2024-01-22 21:02:24.032 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2024-01-22 21:02:24.032 ServerApp] Jupyter Server 2.8.0 is running at:
[I 2024-01-22 21:02:24.032 ServerApp] http://573776c4caf6:8888/user/yassine/lab?token=...
[I 2024-01-22 21:02:24.032 ServerApp]     http://127.0.0.1:8888/user/yassine/lab?token=...
[I 2024-01-22 21:02:24.032 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-01-22 21:02:24.481 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-01-22 21:02:28.193 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.27ms
[I 2024-01-22 21:02:29.214 ServerApp] 302 GET /user/yassine -> /user/yassine/lab? (@::ffff:10.69.150.160) 1.08ms
[I 2024-01-22 21:02:29.222 ServerApp] 302 GET /user/yassine/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-yassine&redirect_uri=%2Fuser%2Fyassine%2Foauth_callback&response_type=code&state=[secret] (@::ffff:10.69.150.160) 2.41ms
[I 2024-01-22 21:02:29.296 ServerApp] Logged-in user {'kind': 'user', 'name': 'yassine', 'admin': False, 'groups': [], 'session_id': '0cf49fdccda7437fa2b2739fbe74ab23', 'scopes': ['access:servers!server=yassine/', 'read:users:groups!user=yassine', 'read:users:name!user=yassine']}
[I 2024-01-22 21:02:29.298 ServerApp] 302 GET /user/yassine/oauth_callback?code=[secret]&state=[secret] -> /user/yassine/lab? (@::ffff:10.69.150.160) 37.73ms
[I 2024-01-22 21:02:29.314 ServerApp] 200 GET /user/yassine/lab? (yassine@::ffff:10.69.150.160) 9.26ms
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 2024-01-22 21:02:30.144 ServerApp] 200 GET /user/yassine/api/kernelspecs?1705957349930 (yassine@::ffff:10.69.150.160) 205.44ms
[I 2024-01-22 21:02:30.150 ServerApp] 200 GET /user/yassine/api/me?1705957349931 (yassine@::ffff:10.69.150.160) 4.87ms
[I 2024-01-22 21:02:30.220 ServerApp] 200 GET /user/yassine/lab/api/settings?1705957349943 (yassine@::ffff:10.69.150.160) 74.24ms
[I 2024-01-22 21:02:30.221 ServerApp] 101 GET /user/yassine/api/events/subscribe?token=[secret] (yassine@::ffff:10.69.150.160) 73.62ms
[I 2024-01-22 21:02:30.222 ServerApp] 200 GET /user/yassine/api/kernels?1705957349952 (yassine@::ffff:10.69.150.160) 74.03ms
[I 2024-01-22 21:02:30.223 ServerApp] 200 GET /user/yassine/api/sessions?1705957349954 (yassine@::ffff:10.69.150.160) 74.83ms
[I 2024-01-22 21:02:30.224 ServerApp] 200 GET /user/yassine/api/terminals?1705957349956 (yassine@::ffff:10.69.150.160) 75.04ms
[I 2024-01-22 21:02:30.227 ServerApp] 200 GET /user/yassine/api/kernelspecs?1705957350143 (yassine@::ffff:10.69.150.160) 1.76ms
[I 2024-01-22 21:02:30.227 ServerApp] 200 GET /user/yassine/api/me?1705957350150 (yassine@::ffff:10.69.150.160) 1.85ms
[I 2024-01-22 21:02:32.269 ServerApp] 200 GET /user/yassine/lab/api/translations/default?1705957352232 (yassine@::ffff:10.69.150.160) 26.38ms
[I 2024-01-22 21:02:32.301 ServerApp] 200 GET /user/yassine/api/config/jupyterlabapputilsextensionannouncements?1705957352284 (yassine@::ffff:10.69.150.160) 7.38ms
[I 2024-01-22 21:02:32.439 ServerApp] 200 GET /user/yassine/lsp/status?1705957352423 (yassine@::ffff:10.69.150.160) 6.85ms
[I 2024-01-22 21:02:32.517 ServerApp] 200 GET /user/yassine/lab/api/translations?1705957352465 (yassine@::ffff:10.69.150.160) 43.11ms
[I 2024-01-22 21:02:32.519 ServerApp] 200 GET /user/yassine/api/contents?content=1&1705957352479 (yassine@::ffff:10.69.150.160) 29.46ms
[I 2024-01-22 21:02:32.623 ServerApp] 200 GET /user/yassine/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1705957352604 (yassine@::ffff:10.69.150.160) 9.34ms
[I 2024-01-22 21:02:32.630 ServerApp] 200 GET /user/yassine/lab/api/settings/@jupyterlab/notebook-extension:panel?1705957352611 (yassine@::ffff:10.69.150.160) 4.99ms
[I 2024-01-22 21:02:32.674 ServerApp] 200 GET /user/yassine/lab/api/workspaces/default?1705957352664 (yassine@::ffff:10.69.150.160) 1.45ms
[I 2024-01-22 21:02:32.726 ServerApp] 200 GET /user/yassine/api/contents?content=1&1705957352710 (yassine@::ffff:10.69.150.160) 6.67ms
[I 2024-01-22 21:02:32.774 ServerApp] 200 GET /user/yassine/lab/api/settings?ids_only=true&1705957352757 (yassine@::ffff:10.69.150.160) 8.35ms
[I 2024-01-22 21:02:32.963 LabApp] Build is up to date
[I 2024-01-22 21:02:32.964 ServerApp] 200 GET /user/yassine/lab/api/build?1705957352316 (yassine@::ffff:10.69.150.160) 639.61ms
[I 2024-01-22 21:02:34.141 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.90ms
[I 2024-01-22 21:02:34.248 ServerApp] 204 PUT /user/yassine/lab/api/workspaces/default?1705957354234 (yassine@::ffff:10.69.150.160) 4.02ms
[I 2024-01-22 21:02:35.251 ServerApp] 204 PUT /user/yassine/lab/api/workspaces/default?1705957355239 (yassine@::ffff:10.69.150.160) 2.24ms
[I 2024-01-22 21:02:40.152 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.64ms
[I 2024-01-22 21:02:41.246 ServerApp] 200 GET /user/yassine/api/kernels?1705957361230 (yassine@::ffff:10.69.150.160) 2.06ms
[I 2024-01-22 21:02:41.249 ServerApp] 200 GET /user/yassine/api/sessions?1705957361231 (yassine@::ffff:10.69.150.160) 1.90ms
[I 2024-01-22 21:02:41.252 ServerApp] 200 GET /user/yassine/api/terminals?1705957361232 (yassine@::ffff:10.69.150.160) 1.79ms
[I 2024-01-22 21:02:46.137 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.73ms
[I 2024-01-22 21:02:52.129 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.49ms
[I 2024-01-22 21:02:58.134 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.71ms
[I 2024-01-22 21:03:04.139 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.70ms
[I 2024-01-22 21:03:10.139 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.59ms
[I 2024-01-22 21:03:16.141 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.71ms
[I 2024-01-22 21:03:22.136 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 2.18ms
[I 2024-01-22 21:03:28.139 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.47ms
[I 2024-01-22 21:03:29.964 ServerApp] 204 PUT /user/yassine/lab/api/workspaces/default?1705957409950 (yassine@::ffff:10.69.150.160) 3.19ms
[I 2024-01-22 21:03:31.420 ServerApp] 200 GET /user/yassine/api/kernelspecs?1705957411402 (yassine@::ffff:10.69.150.160) 4.60ms
[I 2024-01-22 21:03:31.422 ServerApp] 200 GET /user/yassine/api/me?1705957411403 (yassine@::ffff:10.69.150.160) 5.55ms
[I 2024-01-22 21:03:34.130 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.55ms
[I 2024-01-22 21:03:40.146 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.71ms
[I 2024-01-22 21:03:46.135 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.48ms
[I 2024-01-22 21:03:52.147 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.49ms
[I 2024-01-22 21:03:58.128 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.43ms
[I 2024-01-22 21:04:04.136 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.64ms
[I 2024-01-22 21:04:10.124 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.48ms
[I 2024-01-22 21:04:16.139 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.69ms
[I 2024-01-22 21:04:22.131 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.41ms
[I 2024-01-22 21:04:28.118 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.42ms
[I 2024-01-22 21:04:33.249 ServerApp] 200 GET /user/yassine/api/kernelspecs?1705957473233 (yassine@::ffff:10.69.150.160) 3.42ms
[I 2024-01-22 21:04:33.253 ServerApp] 200 GET /user/yassine/api/me?1705957473235 (yassine@::ffff:10.69.150.160) 2.56ms
[I 2024-01-22 21:04:34.121 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.65ms
[I 2024-01-22 21:04:40.131 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.84ms
[I 2024-01-22 21:04:46.141 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.66ms
[I 2024-01-22 21:04:52.136 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.65ms
[I 2024-01-22 21:04:58.134 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.46ms
[I 2024-01-22 21:05:04.120 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.95ms
[I 2024-01-22 21:05:10.133 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 1.44ms
[I 2024-01-22 21:05:16.136 ServerApp] 200 GET /user/yassine/api (@192.168.1.11) 0.69ms
9

Hallo Simon,

your config file works for me as well. I decided to run Jupyterhub on the host, until I found a solution to that problem. Thanks a lot!