Docker images fail if newer than 2023-06-01

We use multiple docker images, e.g. jupyter/minimal-notebook:2023-06-01
all these start and work well.
But all newer images do not start.

My guess: something changed in the images (as I tried lots of different images) and need something new to start?

config:

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.allowed_images = {
  "jupyter/minimal-notebook:2023-06-01" : "jupyter/minimal-notebook:2023-06-01",
  "jupyter/minimal-notebook:2023-08-20" : "jupyter/minimal-notebook:2023-08-20",
   "jupyter/scipy-notebook:2023-06-01" : "jupyter/scipy-notebook:2023-06-01",
   "jupyter/tensorflow-notebook:2023-06-01" : "jupyter/tensorflow-notebook:2023-06-01",
   "jupyter/datascience-notebook:2023-06-01" : "jupyter/datascience-notebook:2023-06-01",
   "jupyter/all-spark-notebook:2023-06-01" : "jupyter/all-spark-notebook:2023-06-01",
   "sgi_dsp_hs2022" : "cr.gitlab.fhnw.ch/ml/courses/dsp/dsp_hs2022:v20220919",
   "sgds_gml_hs2022" : "cr.gitlab.fhnw.ch/ml/courses/gml/gml_trainingcenter:v20220919",
   "sgi_ml_hs2023" : "cr.gitlab.fhnw.ch/ml/courses/ml/ml_hs2023/ml_assignments_hs2023:v20230905"

}

c.Spawner.environment = {'GRANT_SUDO': 'yes'}

c.DockerSpawner.remove = True
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
   "/data/users/{username}": "/home/jovyan/work/persistent"
   }

c.DockerSpawner.read_only_volumes = {
	'/data/readonly': '/data/readonly'
	} 

## Redirect user to server (if running), instead of control panel.
#  Default: True
# c.JupyterHub.redirect_to_server = True
c.JupyterHub.redirect_to_server = False

## Shuts down all user servers on logout
#  Default: False
# c.JupyterHub.shutdown_on_logout = False
# not logged in users should not have running server(s)
c.JupyterHub.shutdown_on_logout = True

## The URL the single-user server should start in.
#c.Spawner.default_url = '/lab' #added for lab, deprecated
c.Spawner.cmd=["jupyter-labhub"]

version: JupyterHub 4.0.2 20230912083027

What errors are you seeing in the JupyterHub and sngleuser container logs?

uses the latest version of the JupyterHub and base-notebook images, and is passing on CI

Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.741 JupyterHub provider:659] Creating oauth client jupyterhub-user-jackie.schindler
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.795 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.844 JupyterHub dockerspawner:1272] Created container jupyter-jackie-2eschindler (id: 955ef4f) from image jupyter/minimal-notebook:2023-08-20
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.845 JupyterHub dockerspawner:1296] Starting container jupyter-jackie-2eschindler (id: 955ef4f)
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.722 JupyterHub log:191] 302 POST /hub/spawn/jackie.schindler?_xsrf=[secret] -> /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.239.241.89) 1008.95ms
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.748 JupyterHub pages:398] jackie.schindler is pending spawn
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.753 JupyterHub log:191] 200 GET /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.239.241.89) 9.20ms
Nov  9 08:28:54 pub094 jupyterhub[917]: [I 2023-11-09 08:28:54.725 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:28:54 pub094 jupyterhub[917]: [W 2023-11-09 08:28:54.726 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:28:54 pub094 jupyterhub[917]: Task exception was never retrieved
Nov  9 08:28:54 pub094 jupyterhub[917]: future: <Task finished name='Task-17889' coro=<BaseHandler.spawn_single_user() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:871> exception=HTTPError()>
Nov  9 08:28:54 pub094 jupyterhub[917]: Traceback (most recent call last):
Nov  9 08:28:54 pub094 jupyterhub[917]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1078, in spawn_single_user
Nov  9 08:28:54 pub094 jupyterhub[917]:     await gen.with_timeout(
Nov  9 08:28:54 pub094 jupyterhub[917]: asyncio.exceptions.TimeoutError: Timeout
Nov  9 08:28:54 pub094 jupyterhub[917]: During handling of the above exception, another exception occurred:
Nov  9 08:28:54 pub094 jupyterhub[917]: Traceback (most recent call last):
Nov  9 08:28:54 pub094 jupyterhub[917]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1112, in spawn_single_user
Nov  9 08:28:54 pub094 jupyterhub[917]:     raise web.HTTPError(
Nov  9 08:28:54 pub094 jupyterhub[917]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=0]. The logs for jackie.schindler may contain details.)
Nov  9 08:29:15 pub094 jupyterhub[917]: [I 2023-11-09 08:29:15.320 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:29:15 pub094 jupyterhub[917]: [W 2023-11-09 08:29:15.321 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:29:16 pub094 jupyterhub[917]: [W 2023-11-09 08:29:16.414 JupyterHub user:933] jackie.schindler's server never showed up at http://127.0.0.1:32768/user/jackie.schindler/ after 30 seconds. Giving up.
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]:     Common causes of this timeout, and debugging tips:
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]:     1. The server didn't finish starting,
Nov  9 08:29:16 pub094 jupyterhub[917]:        or it crashed due to a configuration issue.
Nov  9 08:29:16 pub094 jupyterhub[917]:        Check the single-user server's logs for hints at what needs fixing.
Nov  9 08:29:16 pub094 jupyterhub[917]:     2. The server started, but is not accessible at the specified URL.
Nov  9 08:29:16 pub094 jupyterhub[917]:        This may be a configuration issue specific to your chosen Spawner.
Nov  9 08:29:16 pub094 jupyterhub[917]:        Check the single-user server logs and resource to make sure the URL
Nov  9 08:29:16 pub094 jupyterhub[917]:        is correct and accessible from the Hub.
Nov  9 08:29:16 pub094 jupyterhub[917]:     3. (unlikely) Everything is working, but the server took too long to respond.
Nov  9 08:29:16 pub094 jupyterhub[917]:        To fix: increase `Spawner.http_timeout` configuration
Nov  9 08:29:16 pub094 jupyterhub[917]:        to a number of seconds that is enough for servers to become responsive.
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]: [I 2023-11-09 08:29:16.416 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:29:16 pub094 jupyterhub[917]: [W 2023-11-09 08:29:16.416 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:29:16 pub094 jupyterhub[917]: [E 2023-11-09 08:29:16.439 JupyterHub gen:630] Exception in Future <Task finished name='Task-17890' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://127.0.0.1:32768/user/jackie.schindler/ didn't respond in 30 seconds")> after timeout
Nov  9 08:29:16 pub094 jupyterhub[917]:     Traceback (most recent call last):
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
Nov  9 08:29:16 pub094 jupyterhub[917]:         future.result()
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
Nov  9 08:29:16 pub094 jupyterhub[917]:         await spawn_future
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 914, in spawn
Nov  9 08:29:16 pub094 jupyterhub[917]:         await self._wait_up(spawner)
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 958, in _wait_up
Nov  9 08:29:16 pub094 jupyterhub[917]:         raise e
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 928, in _wait_up
Nov  9 08:29:16 pub094 jupyterhub[917]:         resp = await server.wait_up(
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
Nov  9 08:29:16 pub094 jupyterhub[917]:         re = await exponential_backoff(
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
Nov  9 08:29:16 pub094 jupyterhub[917]:         raise asyncio.TimeoutError(fail_message)
Nov  9 08:29:16 pub094 jupyterhub[917]:     asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:32768/user/jackie.schindler/ didn't respond in 30 seconds
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]: [I 2023-11-09 08:29:16.445 JupyterHub log:191] 200 GET /hub/api/users/jackie.schindler/server/progress?_xsrf=[secret] (jackie.schindler@10.239.241.89) 30316.76ms

my jupyterhub is native in a venv environnement like this:

We only speak about the notebook containers. Jupyterhub is not in a container.
And the containers up to 2023-06-01 are working well.

Can you turn on debug logging and show us the JupyterHub logs? Can you also share the logs for the failed jupyter-jackie-2eschindler container?

Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.552 JupyterHub log:191] 302 GET / -> /hub/ (@10.228.82.183) 1.02ms
Nov 10 09:00:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:16.573 JupyterHub base:344] Refreshing auth for jackie.schindler
Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.578 JupyterHub log:191] 302 GET /hub/ -> /hub/home (jackie.schindler@10.228.82.183) 17.04ms
Nov 10 09:00:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:16.590 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for jackie.schindler:
Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.620 JupyterHub log:191] 200 GET /hub/home (jackie.schindler@10.228.82.183) 31.94ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.005 JupyterHub log:191] 200 GET /hub/static/js/home.js?v=20231110085916 (@10.228.82.183) 4.48ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.025 JupyterHub log:191] 200 GET /hub/static/components/moment/moment.js?v=20231110085916 (@10.228.82.183) 0.98ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.026 JupyterHub log:191] 200 GET /hub/static/js/jhapi.js?v=20231110085916 (@10.228.82.183) 0.67ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.042 JupyterHub log:191] 200 GET /hub/static/js/utils.js?v=20231110085916 (@10.228.82.183) 0.62ms
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub scopes:877] Checking access to /hub/spawn/jackie.schindler via scope servers
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub scopes:677] Unrestricted access to /hub/spawn/jackie.schindler via servers
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub pages:209] Serving options form for jackie.schindler
Nov 10 09:00:31 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:31.560 JupyterHub log:191] 200 GET /hub/spawn/jackie.schindler (jackie.schindler@10.228.82.183) 10.25ms
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub scopes:877] Checking access to /hub/spawn/jackie.schindler via scope servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub scopes:677] Unrestricted access to /hub/spawn/jackie.schindler via servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub pages:257] Triggering spawn with supplied form options for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:961] Initiating spawn for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:965] 0/100 concurrent spawns
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:970] 0 active servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:40.929 JupyterHub provider:659] Creating oauth client jupyterhub-user-jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.942 JupyterHub user:794] Calling Spawner.start for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.964 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:00:40 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:40.965 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.966 JupyterHub dockerspawner:1148] Starting host with config: {'auto_remove': True, 'binds': {'/data/users/jackie-2eschindler': {'bind': '/home/jovyan/work/persistent', 'mode': 'rw'}, '/data/readonly': {'bind': '/data/readonly', 'mode': 'ro'}}, 'links': {}, 'mounts': [], 'mem_limit': 0, 'cpu_period': 100000, 'cpu_quota': 0, 'port_bindings': {8888: ('127.0.0.1',)}, 'network_mode': 'bridge'}
Nov 10 09:00:40 pub094 systemd[1]: var-lib-docker-overlay2-3e9e14c5f37a10e46647c7981c09b93816d020946ffbc7cce547402c9a1695b0\x2dinit-merged.mount: Deactivated successfully.
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.000 JupyterHub dockerspawner:1272] Created container jupyter-jackie-2eschindler (id: 995472b) from image jupyter/minimal-notebook:2023-08-20
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.000 JupyterHub dockerspawner:1296] Starting container jupyter-jackie-2eschindler (id: 995472b)
Nov 10 09:00:41 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:41 pub094 systemd-timesyncd[840]: Network configuration changed, trying to establish connection.
Nov 10 09:00:41 pub094 systemd-networkd[863]: veth58581e9: Link UP
Nov 10 09:00:41 pub094 kernel: [169224.580790] docker0: port 1(veth58581e9) entered blocking state
Nov 10 09:00:41 pub094 kernel: [169224.580795] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:41 pub094 kernel: [169224.580927] device veth58581e9 entered promiscuous mode
Nov 10 09:00:41 pub094 systemd-udevd[55579]: Using default interface naming scheme 'v249'.
Nov 10 09:00:41 pub094 systemd-udevd[55580]: Using default interface naming scheme 'v249'.
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056662353+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056801181+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056813908+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.057319307+01:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 pid=55646 runtime=io.containerd.runc.v2
Nov 10 09:00:41 pub094 systemd[1]: Started libcontainer container 995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.
Nov 10 09:00:41 pub094 kernel: [169224.741395] eth0: renamed from vethd2eab65
Nov 10 09:00:41 pub094 systemd-networkd[863]: veth58581e9: Gained carrier
Nov 10 09:00:41 pub094 systemd-networkd[863]: docker0: Gained carrier
Nov 10 09:00:41 pub094 kernel: [169224.761345] IPv6: ADDRCONF(NETDEV_CHANGE): veth58581e9: link becomes ready
Nov 10 09:00:41 pub094 kernel: [169224.761432] docker0: port 1(veth58581e9) entered blocking state
Nov 10 09:00:41 pub094 kernel: [169224.761439] docker0: port 1(veth58581e9) entered forwarding state
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.253 JupyterHub spawner:1384] Polling subprocess every 30s
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.910 JupyterHub log:191] 302 POST /hub/spawn/jackie.schindler?_xsrf=[secret] -> /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.228.82.183) 1003.94ms
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.925 JupyterHub scopes:877] Checking access to /hub/spawn-pending/jackie.schindler via scope servers
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.925 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/jackie.schindler via servers
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.925 JupyterHub pages:398] jackie.schindler is pending spawn
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.928 JupyterHub log:191] 200 GET /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.228.82.183) 8.23ms
Nov 10 09:00:42 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:42.023 JupyterHub scopes:877] Checking access to /hub/api/users/jackie.schindler/server/progress via scope read:servers
Nov 10 09:00:42 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:42.024 JupyterHub scopes:677] Unrestricted access to /hub/api/users/jackie.schindler/server/progress via read:servers
Nov 10 09:00:42 pub094 systemd-networkd[863]: veth58581e9: Gained IPv6LL
Nov 10 09:00:43 pub094 systemd[1]: docker-995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.scope: Deactivated successfully.
Nov 10 09:00:43 pub094 systemd[1]: docker-995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.scope: Consumed 2.610s CPU time.
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660032178+01:00" level=info msg="shim disconnected" id=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660129952+01:00" level=warning msg="cleaning up after shim disconnected" id=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 namespace=moby
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660155566+01:00" level=info msg="cleaning up dead shim"
Nov 10 09:00:43 pub094 dockerd[1047]: time="2023-11-10T09:00:43.660106018+01:00" level=info msg="ignoring event" container=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.669391191+01:00" level=warning msg="cleanup warnings time=\"2023-11-10T09:00:43+01:00\" level=info msg=\"starting signal loop\" namespace=moby pid=55722 runtime=io.containerd.runc.v2\n"
Nov 10 09:00:43 pub094 systemd-networkd[863]: veth58581e9: Lost carrier
Nov 10 09:00:43 pub094 kernel: [169227.315677] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 kernel: [169227.316159] vethd2eab65: renamed from eth0
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:43 pub094 systemd-networkd[863]: veth58581e9: Link DOWN
Nov 10 09:00:43 pub094 kernel: [169227.365192] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 kernel: [169227.365908] device veth58581e9 left promiscuous mode
Nov 10 09:00:43 pub094 kernel: [169227.365912] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: ERROR:Unknown interface index 6 seen even after reload
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: ERROR:Unknown interface index 6 seen even after reload
Nov 10 09:00:43 pub094 systemd[1]: run-docker-netns-24fa39d30210.mount: Deactivated successfully.
Nov 10 09:00:43 pub094 systemd[1]: var-lib-docker-overlay2-3e9e14c5f37a10e46647c7981c09b93816d020946ffbc7cce547402c9a1695b0-merged.mount: Deactivated successfully.
Nov 10 09:00:44 pub094 systemd-networkd[863]: docker0: Lost carrier
Nov 10 09:00:50 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:50.911 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:00:50 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:50.914 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:00:50 pub094 jupyterhub[55461]: [W 2023-11-10 09:00:50.914 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:00:50 pub094 jupyterhub[55461]: Task exception was never retrieved
Nov 10 09:00:50 pub094 jupyterhub[55461]: future: <Task finished name='Task-44' coro=<BaseHandler.spawn_single_user() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:871> exception=HTTPError()>
Nov 10 09:00:50 pub094 jupyterhub[55461]: Traceback (most recent call last):
Nov 10 09:00:50 pub094 jupyterhub[55461]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1078, in spawn_single_user
Nov 10 09:00:50 pub094 jupyterhub[55461]:     await gen.with_timeout(
Nov 10 09:00:50 pub094 jupyterhub[55461]: asyncio.exceptions.TimeoutError: Timeout
Nov 10 09:00:50 pub094 jupyterhub[55461]: During handling of the above exception, another exception occurred:
Nov 10 09:00:50 pub094 jupyterhub[55461]: Traceback (most recent call last):
Nov 10 09:00:50 pub094 jupyterhub[55461]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1112, in spawn_single_user
Nov 10 09:00:50 pub094 jupyterhub[55461]:     raise web.HTTPError(
Nov 10 09:00:50 pub094 jupyterhub[55461]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=0]. The logs for jackie.schindler may contain details.)
Nov 10 09:01:11 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:11.257 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:01:11 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:11.259 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:01:11 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:11.259 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:12.003 JupyterHub user:933] jackie.schindler's server never showed up at http://127.0.0.1:32769/user/jackie.schindler/ after 30 seconds. Giving up.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]:     Common causes of this timeout, and debugging tips:
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]:     1. The server didn't finish starting,
Nov 10 09:01:12 pub094 jupyterhub[55461]:        or it crashed due to a configuration issue.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        Check the single-user server's logs for hints at what needs fixing.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     2. The server started, but is not accessible at the specified URL.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        This may be a configuration issue specific to your chosen Spawner.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        Check the single-user server logs and resource to make sure the URL
Nov 10 09:01:12 pub094 jupyterhub[55461]:        is correct and accessible from the Hub.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     3. (unlikely) Everything is working, but the server took too long to respond.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        To fix: increase `Spawner.http_timeout` configuration
Nov 10 09:01:12 pub094 jupyterhub[55461]:        to a number of seconds that is enough for servers to become responsive.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.003 JupyterHub user:982] Stopping jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.003 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:01:12 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:12.005 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:01:12 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:12.005 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.016 JupyterHub user:1002] Deleting oauth client jupyterhub-user-jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.024 JupyterHub user:1005] Finished stopping jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [E 2023-11-10 09:01:12.031 JupyterHub gen:630] Exception in Future <Task finished name='Task-45' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://127.0.0.1:32769/user/jackie.schindler/ didn't respond in 30 seconds")> after timeout
Nov 10 09:01:12 pub094 jupyterhub[55461]:     Traceback (most recent call last):
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
Nov 10 09:01:12 pub094 jupyterhub[55461]:         future.result()
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
Nov 10 09:01:12 pub094 jupyterhub[55461]:         await spawn_future
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 914, in spawn
Nov 10 09:01:12 pub094 jupyterhub[55461]:         await self._wait_up(spawner)
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 958, in _wait_up
Nov 10 09:01:12 pub094 jupyterhub[55461]:         raise e
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 928, in _wait_up
Nov 10 09:01:12 pub094 jupyterhub[55461]:         resp = await server.wait_up(
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
Nov 10 09:01:12 pub094 jupyterhub[55461]:         re = await exponential_backoff(
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
Nov 10 09:01:12 pub094 jupyterhub[55461]:         raise asyncio.TimeoutError(fail_message)
Nov 10 09:01:12 pub094 jupyterhub[55461]:     asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:32769/user/jackie.schindler/ didn't respond in 30 seconds
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:12.032 JupyterHub log:191] 200 GET /hub/api/users/jackie.schindler/server/progress?_xsrf=[secret] (jackie.schindler@10.228.82.183) 30012.70ms
Nov 10 09:04:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:04:16.858 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
Nov 10 09:04:16 pub094 jupyterhub[55464]: 09:04:16.860 [ConfigProxy] #033[32minfo#033[39m: 200 GET /api/routes
Nov 10 09:04:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:04:16.861 JupyterHub proxy:392] Checking routes

That is all the logs I have.
I have not found any container logs.

What do you get with docker logs jupyter-jackie-2eschindler?

Error response from daemon: No such container: jupyter-jackie-2eschindler

Ah, that’s hard to know without single user container logs. Could you maybe try spawning a single user server and in another terminal have watch -n1 docker logs jupyter-jackie-2eschindler and one more terminal watch -n1 docker ps -aq?

Yes, there is a short “flash” of messages and then they are gone one second later.
docker shows a container being there for 1 second

Maybe you can redirect the logs in the watch command to a file so that we can see why container crashes straight after it has launched?

[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[W 2023-11-10 10:07:51.292 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.307 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0099s to import
[W 2023-11-10 10:07:51.307 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.311 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0041s to import
[I 2023-11-10 10:07:51.312 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 10:07:51.314 SingleUserLabApp utils:79] Package nbclassic took 0.0018s to import
[W 2023-11-10 10:07:51.316 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.317 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 10:07:51.318 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 10:07:51.318 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.319 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 10:07:51.322 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[W 2023-11-10 10:07:51.323 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.323 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 10:07:51.327 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.327 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 10:07:51.331 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.331 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 10:07:51.334 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.335 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[I 2023-11-10 10:07:51.517 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[W 2023-11-10 10:07:51.521 SingleUserLabApp serverapp:1909] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2023-11-10 10:07:51.533 SingleUserLabApp manager:364] notebook_shim | extension was successfully loaded.
[I 2023-11-10 10:07:51.534 SingleUserLabApp manager:364] jupyter_lsp | extension was successfully loaded.
[I 2023-11-10 10:07:51.535 SingleUserLabApp manager:364] jupyter_server_terminals | extension was successfully loaded.
[I 2023-11-10 10:07:51.536 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2023-11-10 10:07:51.536 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-11-10 10:07:51.537 LabApp] Extension Manager is 'pypi'.
[I 2023-11-10 10:07:51.539 SingleUserLabApp manager:364] jupyterlab | extension was successfully loaded.
[I 2023-11-10 10:07:51.542 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 10:07:51.544 SingleUserLabApp manager:364] notebook | extension was successfully loaded.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 32, in import_item
    pak = getattr(module, obj)
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'notebook.base.handlers' has no attribute 'IPythonHandler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-labhub", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyterlab/labhubapp.py", line 41, in main
    return SingleUserLabApp.launch_instance(argv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 285, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1042, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 955, in initialize
    _patch_app_base_handlers(self)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 866, in _patch_app_base_handlers
    base_handlers.append(import_item(base_handler_name))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 34, in import_item
    raise ImportError("No module named %s" % obj) from e
ImportError: No module named IPythonHandler
[I 2023-11-10 10:07:52.105 SingleUserLabApp manager:299] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

so, got the docker logs

[I 2023-11-10 10:07:51.542 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 10:07:51.544 SingleUserLabApp manager:364] notebook | extension was successfully loaded.

I guess these lines are culprits. I dont think image jupyter/minimal-notebook:2023-08-20 loads nbclassic extension. I guess your “persisted” Jupyter config files on your docker volumes are loading this extension. Could you relaunch the spawner with debug file c.Spawner.debug = True to see what all files single user server is using to load configuration?

[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:198] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:902] Looking for jupyter_server_config in /etc/jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:923] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.625 SingleUserLabApp application:923] Loaded config file: /home/jovyan/.jupyter/jupyter_server_config.py
[W 2023-11-10 12:16:23.625 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.627 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
[I 2023-11-10 12:16:23.667 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0309s to import
[W 2023-11-10 12:16:23.667 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0106s to import
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 12:16:23.684 SingleUserLabApp utils:79] Package nbclassic took 0.0058s to import
[W 2023-11-10 12:16:23.690 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.690 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 12:16:23.693 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 12:16:23.693 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.693 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 12:16:23.697 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.697 SingleUserLabApp application:457] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'ServerApp': {'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[W 2023-11-10 12:16:23.698 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.698 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 12:16:23.702 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.702 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 12:16:23.706 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.706 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 12:16:23.710 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.710 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:198] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:902] Looking for jupyter_server_config in /etc/jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:923] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.625 SingleUserLabApp application:923] Loaded config file: /home/jovyan/.jupyter/jupyter_server_config.py
[W 2023-11-10 12:16:23.625 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.627 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
[I 2023-11-10 12:16:23.667 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0309s to import
[W 2023-11-10 12:16:23.667 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0106s to import
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 12:16:23.684 SingleUserLabApp utils:79] Package nbclassic took 0.0058s to import
[W 2023-11-10 12:16:23.690 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.690 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 12:16:23.693 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 12:16:23.693 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.693 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 12:16:23.697 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.697 SingleUserLabApp application:457] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'ServerApp': {'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[W 2023-11-10 12:16:23.698 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.698 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 12:16:23.702 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.702 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 12:16:23.706 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.706 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 12:16:23.710 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.710 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[D 2023-11-10 12:16:24.130 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.130 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.131 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.131 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyter-lsp-notebook.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.132 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.132 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2023-11-10 12:16:24.132 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[W 2023-11-10 12:16:24.136 SingleUserLabApp serverapp:1909] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2023-11-10 12:16:24.157 SingleUserLabApp manager:364] notebook_shim | extension was successfully loaded.
[D 2023-11-10 12:16:24.158 SingleUserLabApp serverextension:63] [lsp] rootUri will be file:///home/jovyan/work
[D 2023-11-10 12:16:24.158 SingleUserLabApp serverextension:67] [lsp] virtualDocumentsUri will be file:///home/jovyan/work/.virtual_documents
[I 2023-11-10 12:16:24.159 SingleUserLabApp manager:364] jupyter_lsp | extension was successfully loaded.
[I 2023-11-10 12:16:24.160 SingleUserLabApp manager:364] jupyter_server_terminals | extension was successfully loaded.
[I 2023-11-10 12:16:24.164 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2023-11-10 12:16:24.164 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-11-10 12:16:24.164 LabApp] Extension Manager is 'pypi'.
[I 2023-11-10 12:16:24.166 SingleUserLabApp manager:364] jupyterlab | extension was successfully loaded.
[I 2023-11-10 12:16:24.173 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 12:16:24.175 SingleUserLabApp manager:364] notebook | extension was successfully loaded.
[D 2023-11-10 12:16:24.177 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/usr/local/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/opt/conda/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/home/jovyan/.jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 32, in import_item
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyter-lsp-notebook.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
    pak = getattr(module, obj)
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'notebook.base.handlers' has no attribute 'IPythonHandler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-labhub", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyterlab/labhubapp.py", line 41, in main
[D 2023-11-10 12:16:24.192 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
    return SingleUserLabApp.launch_instance(argv)
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 285, in launch_instance
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
    return super().launch_instance(argv=argv, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1042, in launch_instance
[D 2023-11-10 12:16:24.194 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:24.195 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
    app.initialize(argv)
[D 2023-11-10 12:16:24.195 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 955, in initialize
    _patch_app_base_handlers(self)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 866, in _patch_app_base_handlers
    base_handlers.append(import_item(base_handler_name))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 34, in import_item
    raise ImportError("No module named %s" % obj) from e
ImportError: No module named IPythonHandler

something like that?

Could you try with c.Spawner.cmd=["jupyterhub-singleuser"]?

Thanks a 1000 times, now it works!!

2 Likes