Docker images fail if newer than 2023-06-01

Jackie_Schindler · November 8, 2023, 8:24am

We use multiple docker images, e.g. jupyter/minimal-notebook:2023-06-01
all these start and work well.
But all newer images do not start.

My guess: something changed in the images (as I tried lots of different images) and need something new to start?

config:

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.allowed_images = {
  "jupyter/minimal-notebook:2023-06-01" : "jupyter/minimal-notebook:2023-06-01",
  "jupyter/minimal-notebook:2023-08-20" : "jupyter/minimal-notebook:2023-08-20",
   "jupyter/scipy-notebook:2023-06-01" : "jupyter/scipy-notebook:2023-06-01",
   "jupyter/tensorflow-notebook:2023-06-01" : "jupyter/tensorflow-notebook:2023-06-01",
   "jupyter/datascience-notebook:2023-06-01" : "jupyter/datascience-notebook:2023-06-01",
   "jupyter/all-spark-notebook:2023-06-01" : "jupyter/all-spark-notebook:2023-06-01",
   "sgi_dsp_hs2022" : "cr.gitlab.fhnw.ch/ml/courses/dsp/dsp_hs2022:v20220919",
   "sgds_gml_hs2022" : "cr.gitlab.fhnw.ch/ml/courses/gml/gml_trainingcenter:v20220919",
   "sgi_ml_hs2023" : "cr.gitlab.fhnw.ch/ml/courses/ml/ml_hs2023/ml_assignments_hs2023:v20230905"

}

c.Spawner.environment = {'GRANT_SUDO': 'yes'}

c.DockerSpawner.remove = True
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {
   "/data/users/{username}": "/home/jovyan/work/persistent"
   }

c.DockerSpawner.read_only_volumes = {
	'/data/readonly': '/data/readonly'
	} 

## Redirect user to server (if running), instead of control panel.
#  Default: True
# c.JupyterHub.redirect_to_server = True
c.JupyterHub.redirect_to_server = False

## Shuts down all user servers on logout
#  Default: False
# c.JupyterHub.shutdown_on_logout = False
# not logged in users should not have running server(s)
c.JupyterHub.shutdown_on_logout = True

## The URL the single-user server should start in.
#c.Spawner.default_url = '/lab' #added for lab, deprecated
c.Spawner.cmd=["jupyter-labhub"]

version: JupyterHub 4.0.2 20230912083027

manics · November 8, 2023, 12:34pm

What errors are you seeing in the JupyterHub and sngleuser container logs?

github.com

jupyterhub/jupyterhub-deploy-docker/blob/e31f3a5c9e12cb2d3e9fd30d9ec484ddf53c0c31/basic-example/docker-compose.yml

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# JupyterHub docker-compose configuration file
version: "3"

services:
  hub:
    build:
      context: .
      dockerfile: Dockerfile.jupyterhub
      args:
        JUPYTERHUB_VERSION: latest
    restart: always
    image: jupyterhub
    container_name: jupyterhub
    networks:
      - jupyterhub-network
    volumes:
      # The JupyterHub configuration file

This file has been truncated. show original

uses the latest version of the JupyterHub and base-notebook images, and is passing on CI

Jackie_Schindler · November 9, 2023, 7:33am

Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.741 JupyterHub provider:659] Creating oauth client jupyterhub-user-jackie.schindler
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.795 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.844 JupyterHub dockerspawner:1272] Created container jupyter-jackie-2eschindler (id: 955ef4f) from image jupyter/minimal-notebook:2023-08-20
Nov  9 08:28:44 pub094 jupyterhub[917]: [I 2023-11-09 08:28:44.845 JupyterHub dockerspawner:1296] Starting container jupyter-jackie-2eschindler (id: 955ef4f)
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.722 JupyterHub log:191] 302 POST /hub/spawn/jackie.schindler?_xsrf=[secret] -> /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.239.241.89) 1008.95ms
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.748 JupyterHub pages:398] jackie.schindler is pending spawn
Nov  9 08:28:45 pub094 jupyterhub[917]: [I 2023-11-09 08:28:45.753 JupyterHub log:191] 200 GET /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.239.241.89) 9.20ms
Nov  9 08:28:54 pub094 jupyterhub[917]: [I 2023-11-09 08:28:54.725 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:28:54 pub094 jupyterhub[917]: [W 2023-11-09 08:28:54.726 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:28:54 pub094 jupyterhub[917]: Task exception was never retrieved
Nov  9 08:28:54 pub094 jupyterhub[917]: future: <Task finished name='Task-17889' coro=<BaseHandler.spawn_single_user() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:871> exception=HTTPError()>
Nov  9 08:28:54 pub094 jupyterhub[917]: Traceback (most recent call last):
Nov  9 08:28:54 pub094 jupyterhub[917]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1078, in spawn_single_user
Nov  9 08:28:54 pub094 jupyterhub[917]:     await gen.with_timeout(
Nov  9 08:28:54 pub094 jupyterhub[917]: asyncio.exceptions.TimeoutError: Timeout
Nov  9 08:28:54 pub094 jupyterhub[917]: During handling of the above exception, another exception occurred:
Nov  9 08:28:54 pub094 jupyterhub[917]: Traceback (most recent call last):
Nov  9 08:28:54 pub094 jupyterhub[917]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1112, in spawn_single_user
Nov  9 08:28:54 pub094 jupyterhub[917]:     raise web.HTTPError(
Nov  9 08:28:54 pub094 jupyterhub[917]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=0]. The logs for jackie.schindler may contain details.)
Nov  9 08:29:15 pub094 jupyterhub[917]: [I 2023-11-09 08:29:15.320 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:29:15 pub094 jupyterhub[917]: [W 2023-11-09 08:29:15.321 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:29:16 pub094 jupyterhub[917]: [W 2023-11-09 08:29:16.414 JupyterHub user:933] jackie.schindler's server never showed up at http://127.0.0.1:32768/user/jackie.schindler/ after 30 seconds. Giving up.
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]:     Common causes of this timeout, and debugging tips:
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]:     1. The server didn't finish starting,
Nov  9 08:29:16 pub094 jupyterhub[917]:        or it crashed due to a configuration issue.
Nov  9 08:29:16 pub094 jupyterhub[917]:        Check the single-user server's logs for hints at what needs fixing.
Nov  9 08:29:16 pub094 jupyterhub[917]:     2. The server started, but is not accessible at the specified URL.
Nov  9 08:29:16 pub094 jupyterhub[917]:        This may be a configuration issue specific to your chosen Spawner.
Nov  9 08:29:16 pub094 jupyterhub[917]:        Check the single-user server logs and resource to make sure the URL
Nov  9 08:29:16 pub094 jupyterhub[917]:        is correct and accessible from the Hub.
Nov  9 08:29:16 pub094 jupyterhub[917]:     3. (unlikely) Everything is working, but the server took too long to respond.
Nov  9 08:29:16 pub094 jupyterhub[917]:        To fix: increase `Spawner.http_timeout` configuration
Nov  9 08:29:16 pub094 jupyterhub[917]:        to a number of seconds that is enough for servers to become responsive.
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]: [I 2023-11-09 08:29:16.416 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov  9 08:29:16 pub094 jupyterhub[917]: [W 2023-11-09 08:29:16.416 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov  9 08:29:16 pub094 jupyterhub[917]: [E 2023-11-09 08:29:16.439 JupyterHub gen:630] Exception in Future <Task finished name='Task-17890' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://127.0.0.1:32768/user/jackie.schindler/ didn't respond in 30 seconds")> after timeout
Nov  9 08:29:16 pub094 jupyterhub[917]:     Traceback (most recent call last):
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
Nov  9 08:29:16 pub094 jupyterhub[917]:         future.result()
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
Nov  9 08:29:16 pub094 jupyterhub[917]:         await spawn_future
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 914, in spawn
Nov  9 08:29:16 pub094 jupyterhub[917]:         await self._wait_up(spawner)
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 958, in _wait_up
Nov  9 08:29:16 pub094 jupyterhub[917]:         raise e
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 928, in _wait_up
Nov  9 08:29:16 pub094 jupyterhub[917]:         resp = await server.wait_up(
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
Nov  9 08:29:16 pub094 jupyterhub[917]:         re = await exponential_backoff(
Nov  9 08:29:16 pub094 jupyterhub[917]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
Nov  9 08:29:16 pub094 jupyterhub[917]:         raise asyncio.TimeoutError(fail_message)
Nov  9 08:29:16 pub094 jupyterhub[917]:     asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:32768/user/jackie.schindler/ didn't respond in 30 seconds
Nov  9 08:29:16 pub094 jupyterhub[917]:     
Nov  9 08:29:16 pub094 jupyterhub[917]: [I 2023-11-09 08:29:16.445 JupyterHub log:191] 200 GET /hub/api/users/jackie.schindler/server/progress?_xsrf=[secret] (jackie.schindler@10.239.241.89) 30316.76ms

my jupyterhub is native in a venv environnement like this:

github.com

jupyterhub/jupyterhub-the-hard-way/blob/35ddfb49ad81771551c6549696ccec960564d5e4/docs/installation-guide-hard.md

# Install JupyterHub and JupyterLab from the ground up

The combination of [JupyterHub](https://jupyterhub.readthedocs.io) and [JupyterLab](https://jupyterlab.readthedocs.io)
is a great way to make shared computing resources available to a group.

These instructions are a guide for a manual, 'bare metal' install of [JupyterHub](https://jupyterhub.readthedocs.io)
and [JupyterLab](https://jupyterlab.readthedocs.io). This is ideal for running on a single server: build a beast
of a machine and share it within your lab, or use a virtual machine from any VPS or cloud provider.

This guide has similar goals to [The Littlest JupyterHub](https://the-littlest-jupyterhub.readthedocs.io) setup
script. However, instead of bundling all these step for you into one installer, we will perform every step manually.
This makes it easy to customize any part (e.g. if you want to run other services on the same system and need to make them
work together), as well as giving you full control and understanding of your setup.

## Prerequisites

Your own server with administrator (root) access. This could be a local machine, a remotely hosted one, or a cloud instance
or VPS. Each user who will access JupyterHub should have a standard user account on the machine. The install will be done
through the command line - useful if you log into your machine remotely using SSH.

This file has been truncated. show original

We only speak about the notebook containers. Jupyterhub is not in a container.
And the containers up to 2023-06-01 are working well.

manics · November 9, 2023, 5:59pm

Can you turn on debug logging and show us the JupyterHub logs? Can you also share the logs for the failed jupyter-jackie-2eschindler container?

Jackie_Schindler · November 10, 2023, 8:10am

Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.552 JupyterHub log:191] 302 GET / -> /hub/ (@10.228.82.183) 1.02ms
Nov 10 09:00:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:16.573 JupyterHub base:344] Refreshing auth for jackie.schindler
Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.578 JupyterHub log:191] 302 GET /hub/ -> /hub/home (jackie.schindler@10.228.82.183) 17.04ms
Nov 10 09:00:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:16.590 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for jackie.schindler:
Nov 10 09:00:16 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:16.620 JupyterHub log:191] 200 GET /hub/home (jackie.schindler@10.228.82.183) 31.94ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.005 JupyterHub log:191] 200 GET /hub/static/js/home.js?v=20231110085916 (@10.228.82.183) 4.48ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.025 JupyterHub log:191] 200 GET /hub/static/components/moment/moment.js?v=20231110085916 (@10.228.82.183) 0.98ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.026 JupyterHub log:191] 200 GET /hub/static/js/jhapi.js?v=20231110085916 (@10.228.82.183) 0.67ms
Nov 10 09:00:17 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:17.042 JupyterHub log:191] 200 GET /hub/static/js/utils.js?v=20231110085916 (@10.228.82.183) 0.62ms
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub scopes:877] Checking access to /hub/spawn/jackie.schindler via scope servers
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub scopes:677] Unrestricted access to /hub/spawn/jackie.schindler via servers
Nov 10 09:00:31 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:31.554 JupyterHub pages:209] Serving options form for jackie.schindler
Nov 10 09:00:31 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:31.560 JupyterHub log:191] 200 GET /hub/spawn/jackie.schindler (jackie.schindler@10.228.82.183) 10.25ms
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub scopes:877] Checking access to /hub/spawn/jackie.schindler via scope servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub scopes:677] Unrestricted access to /hub/spawn/jackie.schindler via servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.908 JupyterHub pages:257] Triggering spawn with supplied form options for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:961] Initiating spawn for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:965] 0/100 concurrent spawns
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.909 JupyterHub base:970] 0 active servers
Nov 10 09:00:40 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:40.929 JupyterHub provider:659] Creating oauth client jupyterhub-user-jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.942 JupyterHub user:794] Calling Spawner.start for jackie.schindler
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.964 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:00:40 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:40.965 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:00:40 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:40.966 JupyterHub dockerspawner:1148] Starting host with config: {'auto_remove': True, 'binds': {'/data/users/jackie-2eschindler': {'bind': '/home/jovyan/work/persistent', 'mode': 'rw'}, '/data/readonly': {'bind': '/data/readonly', 'mode': 'ro'}}, 'links': {}, 'mounts': [], 'mem_limit': 0, 'cpu_period': 100000, 'cpu_quota': 0, 'port_bindings': {8888: ('127.0.0.1',)}, 'network_mode': 'bridge'}
Nov 10 09:00:40 pub094 systemd[1]: var-lib-docker-overlay2-3e9e14c5f37a10e46647c7981c09b93816d020946ffbc7cce547402c9a1695b0\x2dinit-merged.mount: Deactivated successfully.
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.000 JupyterHub dockerspawner:1272] Created container jupyter-jackie-2eschindler (id: 995472b) from image jupyter/minimal-notebook:2023-08-20
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.000 JupyterHub dockerspawner:1296] Starting container jupyter-jackie-2eschindler (id: 995472b)
Nov 10 09:00:41 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:41 pub094 systemd-timesyncd[840]: Network configuration changed, trying to establish connection.
Nov 10 09:00:41 pub094 systemd-networkd[863]: veth58581e9: Link UP
Nov 10 09:00:41 pub094 kernel: [169224.580790] docker0: port 1(veth58581e9) entered blocking state
Nov 10 09:00:41 pub094 kernel: [169224.580795] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:41 pub094 kernel: [169224.580927] device veth58581e9 entered promiscuous mode
Nov 10 09:00:41 pub094 systemd-udevd[55579]: Using default interface naming scheme 'v249'.
Nov 10 09:00:41 pub094 systemd-udevd[55580]: Using default interface naming scheme 'v249'.
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056662353+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056801181+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.056813908+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Nov 10 09:00:41 pub094 containerd[929]: time="2023-11-10T09:00:41.057319307+01:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 pid=55646 runtime=io.containerd.runc.v2
Nov 10 09:00:41 pub094 systemd[1]: Started libcontainer container 995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.
Nov 10 09:00:41 pub094 kernel: [169224.741395] eth0: renamed from vethd2eab65
Nov 10 09:00:41 pub094 systemd-networkd[863]: veth58581e9: Gained carrier
Nov 10 09:00:41 pub094 systemd-networkd[863]: docker0: Gained carrier
Nov 10 09:00:41 pub094 kernel: [169224.761345] IPv6: ADDRCONF(NETDEV_CHANGE): veth58581e9: link becomes ready
Nov 10 09:00:41 pub094 kernel: [169224.761432] docker0: port 1(veth58581e9) entered blocking state
Nov 10 09:00:41 pub094 kernel: [169224.761439] docker0: port 1(veth58581e9) entered forwarding state
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.253 JupyterHub spawner:1384] Polling subprocess every 30s
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.910 JupyterHub log:191] 302 POST /hub/spawn/jackie.schindler?_xsrf=[secret] -> /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.228.82.183) 1003.94ms
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.925 JupyterHub scopes:877] Checking access to /hub/spawn-pending/jackie.schindler via scope servers
Nov 10 09:00:41 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:41.925 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/jackie.schindler via servers
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.925 JupyterHub pages:398] jackie.schindler is pending spawn
Nov 10 09:00:41 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:41.928 JupyterHub log:191] 200 GET /hub/spawn-pending/jackie.schindler?_xsrf=[secret] (jackie.schindler@10.228.82.183) 8.23ms
Nov 10 09:00:42 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:42.023 JupyterHub scopes:877] Checking access to /hub/api/users/jackie.schindler/server/progress via scope read:servers
Nov 10 09:00:42 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:42.024 JupyterHub scopes:677] Unrestricted access to /hub/api/users/jackie.schindler/server/progress via read:servers
Nov 10 09:00:42 pub094 systemd-networkd[863]: veth58581e9: Gained IPv6LL
Nov 10 09:00:43 pub094 systemd[1]: docker-995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.scope: Deactivated successfully.
Nov 10 09:00:43 pub094 systemd[1]: docker-995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86.scope: Consumed 2.610s CPU time.
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660032178+01:00" level=info msg="shim disconnected" id=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660129952+01:00" level=warning msg="cleaning up after shim disconnected" id=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 namespace=moby
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.660155566+01:00" level=info msg="cleaning up dead shim"
Nov 10 09:00:43 pub094 dockerd[1047]: time="2023-11-10T09:00:43.660106018+01:00" level=info msg="ignoring event" container=995472b5cfe4c5aab55b730f56c6fa728fe926aeb3d1f51050520635a0859d86 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 09:00:43 pub094 containerd[929]: time="2023-11-10T09:00:43.669391191+01:00" level=warning msg="cleanup warnings time=\"2023-11-10T09:00:43+01:00\" level=info msg=\"starting signal loop\" namespace=moby pid=55722 runtime=io.containerd.runc.v2\n"
Nov 10 09:00:43 pub094 systemd-networkd[863]: veth58581e9: Lost carrier
Nov 10 09:00:43 pub094 kernel: [169227.315677] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 kernel: [169227.316159] vethd2eab65: renamed from eth0
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:43 pub094 systemd-networkd[863]: veth58581e9: Link DOWN
Nov 10 09:00:43 pub094 kernel: [169227.365192] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 kernel: [169227.365908] device veth58581e9 left promiscuous mode
Nov 10 09:00:43 pub094 kernel: [169227.365912] docker0: port 1(veth58581e9) entered disabled state
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: ERROR:Unknown interface index 6 seen even after reload
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: WARNING:Unknown index 6 seen, reloading interface list
Nov 10 09:00:43 pub094 networkd-dispatcher[918]: ERROR:Unknown interface index 6 seen even after reload
Nov 10 09:00:43 pub094 systemd[1]: run-docker-netns-24fa39d30210.mount: Deactivated successfully.
Nov 10 09:00:43 pub094 systemd[1]: var-lib-docker-overlay2-3e9e14c5f37a10e46647c7981c09b93816d020946ffbc7cce547402c9a1695b0-merged.mount: Deactivated successfully.
Nov 10 09:00:44 pub094 systemd-networkd[863]: docker0: Lost carrier
Nov 10 09:00:50 pub094 jupyterhub[55461]: [D 2023-11-10 09:00:50.911 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:00:50 pub094 jupyterhub[55461]: [I 2023-11-10 09:00:50.914 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:00:50 pub094 jupyterhub[55461]: [W 2023-11-10 09:00:50.914 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:00:50 pub094 jupyterhub[55461]: Task exception was never retrieved
Nov 10 09:00:50 pub094 jupyterhub[55461]: future: <Task finished name='Task-44' coro=<BaseHandler.spawn_single_user() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:871> exception=HTTPError()>
Nov 10 09:00:50 pub094 jupyterhub[55461]: Traceback (most recent call last):
Nov 10 09:00:50 pub094 jupyterhub[55461]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1078, in spawn_single_user
Nov 10 09:00:50 pub094 jupyterhub[55461]:     await gen.with_timeout(
Nov 10 09:00:50 pub094 jupyterhub[55461]: asyncio.exceptions.TimeoutError: Timeout
Nov 10 09:00:50 pub094 jupyterhub[55461]: During handling of the above exception, another exception occurred:
Nov 10 09:00:50 pub094 jupyterhub[55461]: Traceback (most recent call last):
Nov 10 09:00:50 pub094 jupyterhub[55461]:   File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1112, in spawn_single_user
Nov 10 09:00:50 pub094 jupyterhub[55461]:     raise web.HTTPError(
Nov 10 09:00:50 pub094 jupyterhub[55461]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=0]. The logs for jackie.schindler may contain details.)
Nov 10 09:01:11 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:11.257 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:01:11 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:11.259 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:01:11 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:11.259 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:12.003 JupyterHub user:933] jackie.schindler's server never showed up at http://127.0.0.1:32769/user/jackie.schindler/ after 30 seconds. Giving up.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]:     Common causes of this timeout, and debugging tips:
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]:     1. The server didn't finish starting,
Nov 10 09:01:12 pub094 jupyterhub[55461]:        or it crashed due to a configuration issue.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        Check the single-user server's logs for hints at what needs fixing.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     2. The server started, but is not accessible at the specified URL.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        This may be a configuration issue specific to your chosen Spawner.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        Check the single-user server logs and resource to make sure the URL
Nov 10 09:01:12 pub094 jupyterhub[55461]:        is correct and accessible from the Hub.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     3. (unlikely) Everything is working, but the server took too long to respond.
Nov 10 09:01:12 pub094 jupyterhub[55461]:        To fix: increase `Spawner.http_timeout` configuration
Nov 10 09:01:12 pub094 jupyterhub[55461]:        to a number of seconds that is enough for servers to become responsive.
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.003 JupyterHub user:982] Stopping jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.003 JupyterHub dockerspawner:982] Getting container 'jupyter-jackie-2eschindler'
Nov 10 09:01:12 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:12.005 JupyterHub dockerspawner:988] Container 'jupyter-jackie-2eschindler' is gone
Nov 10 09:01:12 pub094 jupyterhub[55461]: [W 2023-11-10 09:01:12.005 JupyterHub dockerspawner:963] Container not found: jupyter-jackie-2eschindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.016 JupyterHub user:1002] Deleting oauth client jupyterhub-user-jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [D 2023-11-10 09:01:12.024 JupyterHub user:1005] Finished stopping jackie.schindler
Nov 10 09:01:12 pub094 jupyterhub[55461]: [E 2023-11-10 09:01:12.031 JupyterHub gen:630] Exception in Future <Task finished name='Task-45' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:981> exception=TimeoutError("Server at http://127.0.0.1:32769/user/jackie.schindler/ didn't respond in 30 seconds")> after timeout
Nov 10 09:01:12 pub094 jupyterhub[55461]:     Traceback (most recent call last):
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback
Nov 10 09:01:12 pub094 jupyterhub[55461]:         future.result()
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 988, in finish_user_spawn
Nov 10 09:01:12 pub094 jupyterhub[55461]:         await spawn_future
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 914, in spawn
Nov 10 09:01:12 pub094 jupyterhub[55461]:         await self._wait_up(spawner)
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 958, in _wait_up
Nov 10 09:01:12 pub094 jupyterhub[55461]:         raise e
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/user.py", line 928, in _wait_up
Nov 10 09:01:12 pub094 jupyterhub[55461]:         resp = await server.wait_up(
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 289, in wait_for_http_server
Nov 10 09:01:12 pub094 jupyterhub[55461]:         re = await exponential_backoff(
Nov 10 09:01:12 pub094 jupyterhub[55461]:       File "/opt/jupyterhub/lib/python3.10/site-packages/jupyterhub/utils.py", line 237, in exponential_backoff
Nov 10 09:01:12 pub094 jupyterhub[55461]:         raise asyncio.TimeoutError(fail_message)
Nov 10 09:01:12 pub094 jupyterhub[55461]:     asyncio.exceptions.TimeoutError: Server at http://127.0.0.1:32769/user/jackie.schindler/ didn't respond in 30 seconds
Nov 10 09:01:12 pub094 jupyterhub[55461]:     
Nov 10 09:01:12 pub094 jupyterhub[55461]: [I 2023-11-10 09:01:12.032 JupyterHub log:191] 200 GET /hub/api/users/jackie.schindler/server/progress?_xsrf=[secret] (jackie.schindler@10.228.82.183) 30012.70ms
Nov 10 09:04:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:04:16.858 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
Nov 10 09:04:16 pub094 jupyterhub[55464]: 09:04:16.860 [ConfigProxy] #033[32minfo#033[39m: 200 GET /api/routes
Nov 10 09:04:16 pub094 jupyterhub[55461]: [D 2023-11-10 09:04:16.861 JupyterHub proxy:392] Checking routes

That is all the logs I have.
I have not found any container logs.

mahendrapaipuri · November 10, 2023, 8:29am

What do you get with docker logs jupyter-jackie-2eschindler?

Jackie_Schindler · November 10, 2023, 8:43am

Error response from daemon: No such container: jupyter-jackie-2eschindler

mahendrapaipuri · November 10, 2023, 8:57am

Ah, that’s hard to know without single user container logs. Could you maybe try spawning a single user server and in another terminal have watch -n1 docker logs jupyter-jackie-2eschindler and one more terminal watch -n1 docker ps -aq?

Jackie_Schindler · November 10, 2023, 9:11am

Yes, there is a short “flash” of messages and then they are gone one second later.
docker shows a container being there for 1 second

mahendrapaipuri · November 10, 2023, 9:28am

Maybe you can redirect the logs in the watch command to a file so that we can see why container crashes straight after it has launched?

Jackie_Schindler · November 10, 2023, 10:09am

[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 10:07:51.283 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[W 2023-11-10 10:07:51.292 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.307 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0099s to import
[W 2023-11-10 10:07:51.307 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.311 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0041s to import
[I 2023-11-10 10:07:51.312 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 10:07:51.314 SingleUserLabApp utils:79] Package nbclassic took 0.0018s to import
[W 2023-11-10 10:07:51.316 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.317 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 10:07:51.318 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 10:07:51.318 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 10:07:51.319 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 10:07:51.322 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[W 2023-11-10 10:07:51.323 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.323 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 10:07:51.327 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.327 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 10:07:51.331 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.331 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 10:07:51.334 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 10:07:51.335 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[I 2023-11-10 10:07:51.517 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[W 2023-11-10 10:07:51.521 SingleUserLabApp serverapp:1909] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2023-11-10 10:07:51.533 SingleUserLabApp manager:364] notebook_shim | extension was successfully loaded.
[I 2023-11-10 10:07:51.534 SingleUserLabApp manager:364] jupyter_lsp | extension was successfully loaded.
[I 2023-11-10 10:07:51.535 SingleUserLabApp manager:364] jupyter_server_terminals | extension was successfully loaded.
[I 2023-11-10 10:07:51.536 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2023-11-10 10:07:51.536 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-11-10 10:07:51.537 LabApp] Extension Manager is 'pypi'.
[I 2023-11-10 10:07:51.539 SingleUserLabApp manager:364] jupyterlab | extension was successfully loaded.
[I 2023-11-10 10:07:51.542 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 10:07:51.544 SingleUserLabApp manager:364] notebook | extension was successfully loaded.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 32, in import_item
    pak = getattr(module, obj)
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'notebook.base.handlers' has no attribute 'IPythonHandler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-labhub", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyterlab/labhubapp.py", line 41, in main
    return SingleUserLabApp.launch_instance(argv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 285, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1042, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 955, in initialize
    _patch_app_base_handlers(self)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 866, in _patch_app_base_handlers
    base_handlers.append(import_item(base_handler_name))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 34, in import_item
    raise ImportError("No module named %s" % obj) from e
ImportError: No module named IPythonHandler
[I 2023-11-10 10:07:52.105 SingleUserLabApp manager:299] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

so, got the docker logs

mahendrapaipuri · November 10, 2023, 11:22am

[I 2023-11-10 10:07:51.542 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 10:07:51.544 SingleUserLabApp manager:364] notebook | extension was successfully loaded.

I guess these lines are culprits. I dont think image jupyter/minimal-notebook:2023-08-20 loads nbclassic extension. I guess your “persisted” Jupyter config files on your docker volumes are loading this extension. Could you relaunch the spawner with debug file c.Spawner.debug = True to see what all files single user server is using to load configuration?

Jackie_Schindler · November 10, 2023, 12:19pm

[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:198] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:902] Looking for jupyter_server_config in /etc/jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:923] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.625 SingleUserLabApp application:923] Loaded config file: /home/jovyan/.jupyter/jupyter_server_config.py
[W 2023-11-10 12:16:23.625 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.627 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
[I 2023-11-10 12:16:23.667 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0309s to import
[W 2023-11-10 12:16:23.667 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0106s to import
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 12:16:23.684 SingleUserLabApp utils:79] Package nbclassic took 0.0058s to import
[W 2023-11-10 12:16:23.690 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.690 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 12:16:23.693 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 12:16:23.693 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.693 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 12:16:23.697 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.697 SingleUserLabApp application:457] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'ServerApp': {'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[W 2023-11-10 12:16:23.698 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.698 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 12:16:23.702 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.702 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 12:16:23.706 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.706 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 12:16:23.710 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.710 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:547] Starting jupyterhub single-user server version 4.0.2
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 4.0.5
[I 2023-11-10 12:16:23.604 SingleUserLabApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.7.1
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:198] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.620 SingleUserLabApp application:902] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.621 SingleUserLabApp application:902] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:902] Looking for jupyter_server_config in /etc/jupyter
[D 2023-11-10 12:16:23.622 SingleUserLabApp application:923] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2023-11-10 12:16:23.623 SingleUserLabApp application:902] Looking for jupyter_server_config in /home/jovyan/.jupyter
[D 2023-11-10 12:16:23.625 SingleUserLabApp application:923] Loaded config file: /home/jovyan/.jupyter/jupyter_server_config.py
[W 2023-11-10 12:16:23.625 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.626 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.627 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:23.630 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
[I 2023-11-10 12:16:23.667 SingleUserLabApp utils:79] Package jupyter_lsp took 0.0309s to import
[W 2023-11-10 12:16:23.667 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyter_server_terminals took 0.0106s to import
[I 2023-11-10 12:16:23.678 SingleUserLabApp utils:79] Package jupyterlab took 0.0000s to import
[I 2023-11-10 12:16:23.684 SingleUserLabApp utils:79] Package nbclassic took 0.0058s to import
[W 2023-11-10 12:16:23.690 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.690 SingleUserLabApp utils:79] Package notebook took 0.0000s to import
[I 2023-11-10 12:16:23.693 SingleUserLabApp utils:79] Package notebook_shim took 0.0000s to import
[W 2023-11-10 12:16:23.693 SingleUserLabApp utils:92] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2023-11-10 12:16:23.693 SingleUserLabApp manager:344] jupyter_lsp | extension was successfully linked.
[W 2023-11-10 12:16:23.697 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[D 2023-11-10 12:16:23.697 SingleUserLabApp application:457] Config changed: {'FileContentsManager': {'delete_to_trash': False}, 'ServerApp': {'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[W 2023-11-10 12:16:23.698 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.698 SingleUserLabApp manager:344] jupyter_server_terminals | extension was successfully linked.
[W 2023-11-10 12:16:23.702 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.702 SingleUserLabApp manager:344] jupyterlab | extension was successfully linked.
[W 2023-11-10 12:16:23.706 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.706 SingleUserLabApp manager:344] nbclassic | extension was successfully linked.
[W 2023-11-10 12:16:23.710 SingleUserLabApp configurable:200] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2023-11-10 12:16:23.710 SingleUserLabApp manager:344] notebook | extension was successfully linked.
[D 2023-11-10 12:16:24.130 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.130 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.131 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.131 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyter-lsp-notebook.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.132 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.132 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2023-11-10 12:16:24.132 SingleUserLabApp manager:344] notebook_shim | extension was successfully linked.
[W 2023-11-10 12:16:24.136 SingleUserLabApp serverapp:1909] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
[I 2023-11-10 12:16:24.157 SingleUserLabApp manager:364] notebook_shim | extension was successfully loaded.
[D 2023-11-10 12:16:24.158 SingleUserLabApp serverextension:63] [lsp] rootUri will be file:///home/jovyan/work
[D 2023-11-10 12:16:24.158 SingleUserLabApp serverextension:67] [lsp] virtualDocumentsUri will be file:///home/jovyan/work/.virtual_documents
[I 2023-11-10 12:16:24.159 SingleUserLabApp manager:364] jupyter_lsp | extension was successfully loaded.
[I 2023-11-10 12:16:24.160 SingleUserLabApp manager:364] jupyter_server_terminals | extension was successfully loaded.
[I 2023-11-10 12:16:24.164 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2023-11-10 12:16:24.164 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-11-10 12:16:24.164 LabApp] Extension Manager is 'pypi'.
[I 2023-11-10 12:16:24.166 SingleUserLabApp manager:364] jupyterlab | extension was successfully loaded.
[I 2023-11-10 12:16:24.173 SingleUserLabApp manager:364] nbclassic | extension was successfully loaded.
[I 2023-11-10 12:16:24.175 SingleUserLabApp manager:364] notebook | extension was successfully loaded.
[D 2023-11-10 12:16:24.177 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/usr/local/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/opt/conda/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.178 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_config: 
    	/home/jovyan/.jupyter/jupyter_config.json
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 32, in import_item
[D 2023-11-10 12:16:24.179 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyter-lsp-notebook.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
    pak = getattr(module, obj)
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'notebook.base.handlers' has no attribute 'IPythonHandler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-labhub", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyterlab/labhubapp.py", line 41, in main
[D 2023-11-10 12:16:24.192 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
    return SingleUserLabApp.launch_instance(argv)
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.jupyter/jupyter_notebook_config.json
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 285, in launch_instance
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:24.193 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
    return super().launch_instance(argv=argv, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1042, in launch_instance
[D 2023-11-10 12:16:24.194 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2023-11-10 12:16:24.195 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
    app.initialize(argv)
[D 2023-11-10 12:16:24.195 SingleUserLabApp config_manager:92] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.jupyter/jupyter_server_config.json
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 955, in initialize
    _patch_app_base_handlers(self)
  File "/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/mixins.py", line 866, in _patch_app_base_handlers
    base_handlers.append(import_item(base_handler_name))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/traitlets/utils/importstring.py", line 34, in import_item
    raise ImportError("No module named %s" % obj) from e
ImportError: No module named IPythonHandler

something like that?

mahendrapaipuri · November 10, 2023, 1:28pm

Could you try with c.Spawner.cmd=["jupyterhub-singleuser"]?

Jackie_Schindler · November 10, 2023, 1:30pm

Thanks a 1000 times, now it works!!