JupyterHub with SwarmSpawner: failed to load JupyterHubSingleUser server extension

JupyterHub
,
1

I got a Docker Swarm running JupyterHub, but when I launch the stack and spawn a notebook I get the following logs:

$ docker stack deploy -c docker-compose.yaml jupyter
Creating service jupyter_jupyterhub


$ docker service ls
ID             NAME                 MODE         REPLICAS   IMAGE                               PORTS
z4fv4xsqt0a2   jupyter-cretin       replicated   0/1        dsimb/minimal-notebook:x86_64_v4
52aacetru89w   jupyter_jupyterhub   replicated   1/1        dsimb/jupyterhub-swarm-hub:latest   *:8080->8000/tcp


$ docker service logs -f jupyter-cretin
jupyter-cretin.1.q7swfdofmbdu@chili    | Entered start.sh with args: start-notebook.py
jupyter-cretin.1.q7swfdofmbdu@chili    | Running hooks in: /usr/local/bin/start-notebook.d as uid: 0 gid: 0
jupyter-cretin.1.q7swfdofmbdu@chili    | Done running hooks in: /usr/local/bin/start-notebook.d
jupyter-cretin.1.q7swfdofmbdu@chili    | Updated the jovyan user:
jupyter-cretin.1.q7swfdofmbdu@chili    | - username: jovyan       -> cretin
jupyter-cretin.1.q7swfdofmbdu@chili    | - home dir: /home/jovyan -> /home/cretin
jupyter-cretin.1.q7swfdofmbdu@chili    | Update cretin's UID:GID to 2179:2000
jupyter-cretin.1.q7swfdofmbdu@chili    | userdel: group cretin not removed because it is not the primary group of user cretin.
jupyter-cretin.1.q7swfdofmbdu@chili    | Changing working directory to /home/cretin/
jupyter-cretin.1.q7swfdofmbdu@chili    | Granting cretin passwordless sudo rights!
jupyter-cretin.1.q7swfdofmbdu@chili    | Running hooks in: /usr/local/bin/before-notebook.d as uid: 0 gid: 0
jupyter-cretin.1.q7swfdofmbdu@chili    | Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
jupyter-cretin.1.q7swfdofmbdu@chili    | Done running hooks in: /usr/local/bin/before-notebook.d
jupyter-cretin.1.q7swfdofmbdu@chili    | Running as cretin: start-notebook.py
jupyter-cretin.1.q7swfdofmbdu@chili    | [W 2024-03-07 18:20:39.197 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.200 ServerApp] jupyter_lsp | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.203 ServerApp] jupyter_server_terminals | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.203 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.203 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.206 ServerApp] jupyterhub | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [W 2024-03-07 18:20:39.208 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.210 ServerApp] jupyterlab | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [W 2024-03-07 18:20:39.211 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.213 ServerApp] nbclassic | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [W 2024-03-07 18:20:39.215 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.217 ServerApp] notebook | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.444 ServerApp] notebook_shim | extension was successfully linked.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.468 ServerApp] notebook_shim | extension was successfully loaded.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.470 ServerApp] jupyter_lsp | extension was successfully loaded.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:39.471 ServerApp] jupyter_server_terminals | extension was successfully loaded.
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:40.037 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
jupyter-cretin.1.q7swfdofmbdu@chili    | [C 2024-03-07 18:20:49.254 ServerApp] received signal 15, stopping
jupyter-cretin.1.q7swfdofmbdu@chili    | [C 2024-03-07 18:20:49.254 ServerApp] received signal 15, stopping
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:49.255 ServerApp] Shutting down 7 extensions
jupyter-cretin.1.q7swfdofmbdu@chili    | [I 2024-03-07 18:20:49.255 ServerApp] Shutting down 7 extensions
jupyter-cretin.1.q7swfdofmbdu@chili    | [E 2024-03-07 18:20:49.256 JupyterHubSingleUser] Failed to load JupyterHubSingleUser server extension
jupyter-cretin.1.q7swfdofmbdu@chili    |     Traceback (most recent call last):
jupyter-cretin.1.q7swfdofmbdu@chili    |       File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py", line 274, in wrapped
jupyter-cretin.1.q7swfdofmbdu@chili    |         r = f(self, *args, **kwargs)
jupyter-cretin.1.q7swfdofmbdu@chili    |             ^^^^^^^^^^^^^^^^^^^^^^^^
jupyter-cretin.1.q7swfdofmbdu@chili    |       File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py", line 633, in initialize
jupyter-cretin.1.q7swfdofmbdu@chili    |         app.io_loop.run_sync(self.check_hub_version)
jupyter-cretin.1.q7swfdofmbdu@chili    |       File "/opt/conda/lib/python3.11/site-packages/tornado/ioloop.py", line 538, in run_sync
jupyter-cretin.1.q7swfdofmbdu@chili    |         raise TimeoutError("Operation timed out after %s seconds" % timeout)
jupyter-cretin.1.q7swfdofmbdu@chili    |     TimeoutError: Operation timed out after None seconds
[...]

I don’t understand the error, can someone help please ?
Thanks !

2

Strange. My guess is that single user container is not able to communicate with Hub. Can you turn on debug logging and see if there are more details? Also, can you check if single user hub API URL is pointing to correct URL? Generally it is set in env vars and by inspecting env vars of single user container you should be able to retrieve them

3

That’s also what I think because I tested this config on my local network (a vlan with a few machines) and it worked perfectly, I was able to spawn images etc.
Then when I deployed it in “production” mode on the web server, which is on another vlan (but the workers are still in my vlan), I new I would have to ask the IT to open some ports, that’s what I did, and now it works up to this moment, when the singleuser servers have to start.

I have a custom SwarmSpawner class, how do I set the log level to debug in this case ?

class CustomSwarmSpawner(SwarmSpawner):
    [...]
c.JupyterHub.spawner_class = CustomSwarmSpawner

Because I tried to set:

c.SwarmSpawner.debug = True
c.DockerSpawner.debug = True
c.ConfigurableHTTPProxy.debug = True

but it doesn’t change anything for either SwarmSpawner nor DockerSpawner. However I do get the debug logs for ConfigurableHTTPProxy, here they are:

# docker service logs -f jupyter_jupyterhub
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.794 JupyterHub app:2859] Running JupyterHub version 4.0.2
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.794 JupyterHub app:2889] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.794 JupyterHub app:2889] Using Spawner: builtins.CustomSwarmSpawner
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.795 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.841 JupyterHub app:1709] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.878 alembic.runtime.migration migration:213] Context impl SQLiteImpl.
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.878 alembic.runtime.migration migration:216] Will assume non-transactional DDL.
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:48.886 alembic.runtime.migration migration:621] Running stamp_revision  -> 0eee8c825d24
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.023 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.045 JupyterHub roles:238] Adding role admin for User: cretin
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.048 JupyterHub roles:238] Adding role user for User: cretin
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.049 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.070 JupyterHub app:2928] Initialized 0 spawners in 0.002 seconds
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.075 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.076 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.077 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [W 2024-03-08 13:29:49.077 JupyterHub proxy:746] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.077 JupyterHub proxy:750] Starting proxy @ http://:8000
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.387 [ConfigProxy] info: Proxying http://*:8000 to (no default)
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.390 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.655 [ConfigProxy] info: 200 GET /api/routes
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.655 JupyterHub app:3178] Hub API listening on http://jupyterhub:8081/hub/
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.657 [ConfigProxy] info: 200 GET /api/routes
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.658 JupyterHub proxy:477] Adding route for Hub: / => http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.661 [ConfigProxy] info: Adding route / -> http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.661 [ConfigProxy] info: Route added / -> http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:49.662 [ConfigProxy] info: 201 POST /api/routes/
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:49.662 JupyterHub app:3245] JupyterHub is now running at http://:8000
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:51.645 [ConfigProxy] debug: PROXY WEB /hub/ to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:51.662 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:10.0.0.2) 1.78ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:51.664 [ConfigProxy] debug: Not recording activity for status 302 on /
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:51.675 [ConfigProxy] debug: PROXY WEB /hub/login to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:51.716 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:10.0.0.2) 39.30ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:57.547 [ConfigProxy] debug: PROXY WEB /hub/login to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:57.812 JupyterHub base:837] User logged in: cretin
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:57.813 JupyterHub log:191] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (cretin@::ffff:10.0.0.2) 259.32ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:57.815 [ConfigProxy] debug: Not recording activity for status 302 on /
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:57.833 [ConfigProxy] debug: PROXY WEB /hub/ to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [W 2024-03-08 13:29:57.860 JupyterHub configurable:200] Config option `remove` not recognized by `CustomSwarmSpawner`.
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:57.862 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (cretin@::ffff:10.0.0.2) 25.96ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:57.863 [ConfigProxy] debug: Not recording activity for status 302 on /
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:29:57.877 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:29:58.352 JupyterHub log:191] 200 GET /hub/spawn (cretin@::ffff:10.0.0.2) 472.74ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:30:00.996 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:01.136 JupyterHub jupyterhub_config:483] User Mattermost ID: sug6ijojujgzdxmohensu358sw
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:01.235 JupyterHub jupyterhub_config:504] Request successful
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:01.236 JupyterHub jupyterhub_config:508] Channel ID: 5trs4xuxs3ywfnednwxbi5etjy
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:01.550 JupyterHub provider:659] Creating oauth client jupyterhub-user-cretin
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:01.595 JupyterHub dockerspawner:1262] pulling dsimb/gpu-jupyter-v1.6_cuda-12.0_ubuntu-22.04:x86_64_v4
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:02.495 JupyterHub log:191] 302 POST /hub/spawn?_xsrf=[secret] -> /hub/spawn-pending/cretin?_xsrf=[secret] (cretin@::ffff:10.0.0.2) 1494.24ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:30:02.496 [ConfigProxy] debug: Not recording activity for status 302 on /
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:30:02.515 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/cretin to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:02.522 JupyterHub pages:398] cretin is pending spawn
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:02.541 JupyterHub log:191] 200 GET /hub/spawn-pending/cretin?_xsrf=[secret] (cretin@::ffff:10.0.0.2) 23.33ms
jupyter_jupyterhub.1.xyczlxc5gte4@www    | 13:30:02.604 [ConfigProxy] debug: PROXY WEB /hub/api/users/cretin/server/progress to http://jupyterhub:8081
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:02.993 JupyterHub dockerspawner:1033] Service 'jupyter-cretin' is gone
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:03.018 JupyterHub dockerspawner:1311] Created service jupyter-cretin (id: vdyzqmk) from image dsimb/gpu-jupyter-v1.6_cuda-12.0_ubuntu-22.04:x86_64_v4
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [I 2024-03-08 13:30:03.018 JupyterHub dockerspawner:1335] Starting service jupyter-cretin (id: vdyzqmk)
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [W 2024-03-08 13:30:11.495 JupyterHub base:1090] User cretin is slow to start (timeout=10)
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [E 2024-03-08 13:31:18.989 JupyterHub user:884] Unhandled error starting cretin's server: Service jupyter-cretin not running: {'ContainerStatus': {'ContainerID': 'eb4014db8679c37efc4077620060641392d601223c81ed736e3ea2e47e6a580c',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'ExitCode': 1,
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'PID': 0},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Err': 'task: non-zero exit (1): dockerexec: unhealthy container',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Message': 'starting',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'PortStatus': {},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'State': 'failed',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Timestamp': '2024-03-08T13:31:20.899629619Z'}
jupyter_jupyterhub.1.xyczlxc5gte4@www    |     Traceback (most recent call last):
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 798, in spawn
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1343, in start
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         await self.start_object()
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/swarmspawner.py", line 298, in start_object
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         raise RuntimeError(
jupyter_jupyterhub.1.xyczlxc5gte4@www    |     RuntimeError: Service jupyter-cretin not running: {'ContainerStatus': {'ContainerID': 'eb4014db8679c37efc4077620060641392d601223c81ed736e3ea2e47e6a580c',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'ExitCode': 1,
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'PID': 0},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Err': 'task: non-zero exit (1): dockerexec: unhealthy container',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Message': 'starting',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'PortStatus': {},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'State': 'failed',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Timestamp': '2024-03-08T13:31:20.899629619Z'}
jupyter_jupyterhub.1.xyczlxc5gte4@www    |
jupyter_jupyterhub.1.xyczlxc5gte4@www    | [E 2024-03-08 13:31:19.073 JupyterHub gen:630] Exception in Future <Task finished name='Task-34' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/base.py:981> exception=RuntimeError("Service jupyter-cretin not running: {'ContainerStatus': {'ContainerID': 'eb4014db8679c37efc4077620060641392d601223c81ed736e3ea2e47e6a580c',\n                     'ExitCode': 1,\n                     'PID': 0},\n 'Err': 'task: non-zero exit (1): dockerexec: unhealthy container',\n 'Message': 'starting',\n 'PortStatus': {},\n 'State': 'failed',\n 'Timestamp': '2024-03-08T13:31:20.899629619Z'}")> after timeout
jupyter_jupyterhub.1.xyczlxc5gte4@www    |     Traceback (most recent call last):
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 625, in error_callback
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         future.result()
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         await spawn_future
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 902, in spawn
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         raise e
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 798, in spawn
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1343, in start
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         await self.start_object()
jupyter_jupyterhub.1.xyczlxc5gte4@www    |       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/swarmspawner.py", line 298, in start_object
jupyter_jupyterhub.1.xyczlxc5gte4@www    |         raise RuntimeError(
jupyter_jupyterhub.1.xyczlxc5gte4@www    |     RuntimeError: Service jupyter-cretin not running: {'ContainerStatus': {'ContainerID': 'eb4014db8679c37efc4077620060641392d601223c81ed736e3ea2e47e6a580c',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'ExitCode': 1,
jupyter_jupyterhub.1.xyczlxc5gte4@www    |                          'PID': 0},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Err': 'task: non-zero exit (1): dockerexec: unhealthy container',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Message': 'starting',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'PortStatus': {},
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'State': 'failed',
jupyter_jupyterhub.1.xyczlxc5gte4@www    |      'Timestamp': '2024-03-08T13:31:20.899629619Z'}
jupyter_jupyterhub.1.xyczlxc5gte4@www    |
4

Okay, found the bug, I had to change c.JupyterHub.hub_port which changed

1 Like
5

@Gabriel_CRETIN to what value did you change c.JupyterHub.hub_port?

6

I changed it to 8080

1 Like