Switch to Hub/Lab4: Jupyterhub and spawner Jupyterlab container can't communicate properly: "No user identified"

jannefleischer · April 19, 2024, 9:43am

Hi,

I had a working environment with Jupyterhub 3.x and a corresponding jupyterlab 3.x image launched with SystemUserSpawner, which worked very fine, I even got it t work with autofs mounted cifs-share from our local network. Great times.

And then I thought, hey there is Jupyterlab 4 out there, and even nbdiff and other blockers got ported to that, too. So its time to update. Bad choice. Something basic must have been changed, so that my custom images, just upgraded to 4, weren’t runnable anymore. I won’t go into details what I have fixed so far (as I can’t remember very clearly, but switched frm SystemUserSpawner to DockerSpawner and from a bsutom Lab-Image back to the minimalNotebook from the Jupyter Dockerstacks, reset sqlite-db, adjusted config multiple times), and just provide my current setup (which is stripped down to the basics now), so someone might be able give me advice on what to try next.

Basically Jupyterhub runs fine, I can login and launch an image. Then hub can’t get a response from http://172.21.0.3:8888/user/fleischer/ . But The signal 15 (for closing the container) arrives properly there. So whats the shitty config then?

Any help and small hint would help me very much, as I don’t know where to go next. Thank you in advance!

My Dockerfile for my Jupyterhub instance:

FROM quay.io/jupyterhub/jupyterhub:4.1.4
RUN apt-get update && apt-get install -y  git nano
RUN python3 -m pip install --no-cache-dir \
    'docker==7' \
    'dockerspawner' \
    'jupyterhub-nativeauthenticator' \
    'oauthenticator'
CMD ["jupyterhub", "-f", "/srv/jupyterhub/jupyterhub_config.py", "--debug"]

The docker-compose.yml:

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

# JupyterHub docker-compose configuration file
version: "3"

services:
  hub:
    image: jannefleischer/jupyter_ils_env:hub4 #2024-04-12
    restart: unless-stopped
    container_name: jupyterhub
    networks:
      - jupyterhub-network
    volumes:
      - "/home/ils_ubuntu/certs:/certs"
      # The JupyterHub configuration file
      - "./jupyterconf:/srv/jupyterhub/jupyterconf"
      - "./jupyterconf/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.ro.py:ro"
      # Bind Docker socket on the host so we can connect to the daemon from
      # within the container
      - "/var/run/docker.sock:/var/run/docker.sock:rw"
      # Bind Docker volume on host for JupyterHub database and cookie secrets
      - "./jupyterhub-data:/data"
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
      - /etc/gshadow:/etc/gshadow:ro
    ports:
      - "8083:8083"
      - "8080:8080"
      - "8000:8000"
    environment:
      # This username will be a JupyterHub admin
      JUPYTERHUB_ADMIN: admin
      # All containers will join this network
      DOCKER_NETWORK_NAME: jupyterhub-network
      # JupyterHub will spawn this Notebook image for users
      DOCKER_NOTEBOOK_IMAGE: quay.io/jupyter/minimal-notebook:latest
     # Notebook directory inside user image
      DOCKER_NOTEBOOK_DIR: /home/jovyan/work
    entrypoint: ["jupyterhub", "--debug", "-f" , "/srv/jupyterhub/jupyterconf/jupyterhub_config.py"]

volumes:
  jupyterhub-data:

networks:
  jupyterhub-network:
    name: jupyterhub-network

This is my jupyterhub_config.py:

import os
from subprocess import check_call
from pwd import getpwnam
from grp import getgrnam
from oauthenticator.generic import GenericOAuthenticator
from dockerspawner import SystemUserSpawner, DockerSpawner
c = get_config()  # noqa: F821
c.JupyterHub.allow_named_servers=False
c.JupyterHub.authenticator_class = "nativeauthenticator.NativeAuthenticator"
c.NativeAuthenticator.open_signup = True
c.JupyterHub.bind_url = 'https://lab.ils.local'
c.JupyterHub.cookie_secret_file = "/data/jupyterhub_cookie_secret"
c.JupyterHub.db_url = "sqlite:////data/jupyterhub.sqlite"
c.JupyterHub.hub_connect_ip = 'hub'
c.JupyterHub.hub_connect_port = 8083
c.JupyterHub.hub_ip = "jupyterhub"
c.JupyterHub.hub_port = 8083
c.JupyterHub.ip = ''
c.JupyterHub.port = 8080
class MyUidCorrectorDockerSpawner(SystemUserSpawner):
    def uid_for_user(self, user):
        return getpwnam(user).pw_uid
    def gid_for_user(self, group):
        return getgrnam(group).gr_gid
    def get_env(self):
        env = super().get_env()
        env['NB_USER'] = 'jovyan'
        env['NB_UID'] = self.uid_for_user(self.user.name)
        env['NB_GID'] = self.gid_for_user(self.user.name)
        return env
c.JupyterHub.spawner_class = "dockerspawner.DockerSpawner"
c.JupyterHub.ssl_cert = '/certs/srv-geo2-ubuntu.ils.local.crt'
c.JupyterHub.ssl_key = '/certs/srv-geo2-ubuntu.ils.local.key'
c.Spawner.cmd = os.environ.get("DOCKER_SPAWN_CMD", ["jupyterhub-singleuser", "--debug"]) # using the DOCKER_SPAWN_CMD environment variable.
c.DockerSpawner.cmd = c.Spawner.cmd
c.Spawner.debug = True
c.DockerSpawner.debug = c.Spawner.debug
c.Spawner.env_keep = [
    'JUPYTER_CONFIG_DIR'
]
c.DockerSpawner.env_keep = c.Spawner.env_keep
c.Spawner.http_timeout = 2400
c.DockerSpawner.http_timeout = c.Spawner.http_timeout
c.Spawner.ip = '127.0.0.1'
c.DockerSpawner.ip = c.Spawner.ip
c.JupyterHub.name_template = "{prefix}-{username}-{servername}"
c.Spawner.allowed_images = [
  os.environ["DOCKER_NOTEBOOK_IMAGE"],
  'jannefleischer/jupyter_ils_env:lab4',
  'quay.io/jupyter/datascience-notebook:lab-4.1.4',
  'jannefleischer/gds_10.0_ils:latest',
  'darribas/gds:10.0'
]
c.DockerSpawner.allowed_images = c.Spawner.allowed_images
c.JupyterHub.image_whitelist = c.Spawner.allowed_images
c.Spawner.image_homedir_format_string = '/home/{username}/work'
c.Spawner.image_homedir_propagation = 'rshared'
c.Spawner.use_internal_ip = True
c.DockerSpawner.use_internal_ip = c.Spawner.use_internal_ip
c.Spawner.network_name = os.environ["DOCKER_NETWORK_NAME"]
c.DockerSpawner.network_name = c.Spawner.network_name
c.Spawner.volumes = {
    "/home/ils_ubuntu/dockerhome/jupyterhub_stack_dockerspawner/jupyterconf/userconfs": { 'bind': "/home/userconfs", 'mode': 'rw', 'propagation': 'rshared'},
    "/etc/auto.jupyter.d": {'bind': "/etc/auto.jupyter.d", 'mode': 'rw'},
    "/etc/auto.srv-fs1-box": {'bind': "/etc/auto.srv-fs1-box", 'mode': 'ro'},
    "/etc/auto.srv-geo2": {'bind': "/etc/auto.srv-geo2", 'mode': 'ro'},
    "/etc/auto.master.jupyter.d": {'bind': "/etc/auto.master.jupyter.d", 'mode': 'rw'}
}
c.DockerSpawner.volumes = c.Spawner.volumes
c.Spawner.remove = False
c.DockerSpawner.remove = c.Spawner.remove
c.Spawner.port = 8888
c.DockerSpawner.port = c.Spawner.port
def pre_spawn_hook(spawner):
    username = spawner.user.name
    import os
    try:
        import pwd
        spawner.log.debug(spawner.volumes)
        spawner.log.debug(pwd.getpwnam(username).pw_uid)
    except KeyError:
        import subprocess
        spawner.log.warning('users must be created first. we are trying!')
        subprocess.check_call(['groupadd', '-g', os.environ['NB_GID'] , 'jovyan'])
        subprocess.check_call(['useradd', '-ms', '/bin/bash', username, '-g', os.environ['NB_GID']])
    try:
        userconf_dirs = os.listdir("/home/userconfs/")
    except FileNotFoundError:
        os.makedirs("/home/userconfs/")
        userconf_dirs = []
    if username in userconf_dirs:
        pass
    else:
        os.makedirs("/home/userconfs/"+username)
    os.environ['JUPYTER_CONFIG_DIR']="/home/userconfs/"+username
c.Spawner.pre_spawn_hook = pre_spawn_hook
c.DockerSpawner.pre_spawn_hook = c.Spawner.pre_spawn_hook
c.Spawner.start_timeout = 180
c.DockerSpawner.start_timeout = c.Spawner.start_timeout
admin = os.environ.get("JUPYTERHUB_ADMIN")
if admin:
    c.Authenticator.admin_users = [admin]

Logs in additional posts due to discourse limits.

jannefleischer · April 19, 2024, 9:44am

This is the log of jupyterhub:

jupyterhub  | [D 2024-04-19 05:34:54.563 JupyterHub application:908] Looking for /srv/jupyterhub/jupyterconf/jupyterhub_config in /srv/jupyterhub
jupyterhub  | [D 2024-04-19 05:34:54.565 JupyterHub application:929] Loaded config file: /srv/jupyterhub/jupyterconf/jupyterhub_config.py
jupyterhub  | [W 2024-04-19 05:34:54.572 JupyterHub configurable:214] Config option `name_template` not recognized by `JupyterHub`.  Did you mean `template_vars`?
jupyterhub  | [W 2024-04-19 05:34:54.573 JupyterHub configurable:214] Config option `image_whitelist` not recognized by `JupyterHub`.
jupyterhub  | [W 2024-04-19 05:34:54.574 JupyterHub app:693] Both bind_url and ip/port/base_url have been configured. JupyterHub.ip, JupyterHub.port, JupyterHub.base_url are deprecated in JupyterHub 0.9, please use JupyterHub.bind_url instead.
jupyterhub  | [I 2024-04-19 05:34:54.574 JupyterHub app:2885] Running JupyterHub version 4.1.4
jupyterhub  | [I 2024-04-19 05:34:54.574 JupyterHub app:2915] Using Authenticator: nativeauthenticator.nativeauthenticator.NativeAuthenticator
jupyterhub  | [I 2024-04-19 05:34:54.574 JupyterHub app:2915] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-13.0.0
jupyterhub  | [I 2024-04-19 05:34:54.574 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.4
jupyterhub  | [I 2024-04-19 05:34:54.586 JupyterHub app:1683] Loading cookie_secret from /data/jupyterhub_cookie_secret
jupyterhub  | [D 2024-04-19 05:34:54.587 JupyterHub app:1852] Connecting to db: sqlite:////data/jupyterhub.sqlite
jupyterhub  | [D 2024-04-19 05:34:54.619 JupyterHub orm:1029] database schema version found: 0eee8c825d24
jupyterhub  | [W 2024-04-19 05:34:54.669 JupyterHub app:1935] JupyterHub.hub_connect_port is deprecated as of 0.9. Use JupyterHub.hub_connect_url to fully specify the URL for connecting to the Hub.
jupyterhub  | [I 2024-04-19 05:34:54.670 JupyterHub proxy:557] Generating new CONFIGPROXY_AUTH_TOKEN
jupyterhub  | [D 2024-04-19 05:34:54.670 JupyterHub app:2125] Loading roles into database
jupyterhub  | [W 2024-04-19 05:34:54.679 JupyterHub auth:38] 
jupyterhub  |     The shared database session at Authenticator.db is deprecated, and will be removed.
jupyterhub  |     Please manage your own database and connections.
jupyterhub  |     
jupyterhub  |     Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
jupyterhub  |     if you have questions or ideas about direct database needs for your Authenticator.
jupyterhub  |     
jupyterhub  | [I 2024-04-19 05:34:54.683 JupyterHub app:2005] Not using allowed_users. Any authenticated user will be allowed.
jupyterhub  | [D 2024-04-19 05:34:54.687 JupyterHub app:2364] Purging expired APITokens
jupyterhub  | [D 2024-04-19 05:34:54.690 JupyterHub app:2364] Purging expired OAuthCodes
jupyterhub  | [D 2024-04-19 05:34:54.692 JupyterHub app:2200] Loading role assignments from config
jupyterhub  | [D 2024-04-19 05:34:54.706 JupyterHub app:2523] Initializing spawners
jupyterhub  | [D 2024-04-19 05:34:54.707 JupyterHub app:2658] Loaded users:
jupyterhub  |     
jupyterhub  | [I 2024-04-19 05:34:54.707 JupyterHub app:2954] Initialized 0 spawners in 0.002 seconds
jupyterhub  | [W 2024-04-19 05:34:54.709 JupyterHub app:2688] Deleting OAuth client jupyterhub-user-fleischer
jupyterhub  | [I 2024-04-19 05:34:54.735 JupyterHub metrics:279] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
jupyterhub  | [I 2024-04-19 05:34:54.738 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.seven_days
jupyterhub  | [I 2024-04-19 05:34:54.740 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.thirty_days
jupyterhub  | [I 2024-04-19 05:34:54.742 JupyterHub proxy:751] Starting proxy @ https://:8080/
jupyterhub  | [D 2024-04-19 05:34:54.742 JupyterHub proxy:752] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8080', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://hub:8083/hub/error', '--log-level', 'info', '--ssl-key', '/certs/srv-geo2-ubuntu.ils.local.key', '--ssl-cert', '/certs/srv-geo2-ubuntu.ils.local.crt']
jupyterhub  | [D 2024-04-19 05:34:54.743 JupyterHub proxy:671] Writing proxy pid file: jupyterhub-proxy.pid
jupyterhub  | [D 2024-04-19 05:34:54.744 JupyterHub utils:264] Waiting 10s for server at 7735acaea79a:8080
jupyterhub  | [D 2024-04-19 05:34:54.744 JupyterHub utils:111] Server at 7735acaea79a:8080 not ready: [Errno 111] Connection refused
jupyterhub  | [D 2024-04-19 05:34:54.745 JupyterHub utils:264] Waiting 10s for server at 127.0.0.1:8001
jupyterhub  | [D 2024-04-19 05:34:54.745 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
jupyterhub  | [D 2024-04-19 05:34:54.905 JupyterHub utils:111] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
jupyterhub  | [D 2024-04-19 05:34:54.915 JupyterHub utils:111] Server at 7735acaea79a:8080 not ready: [Errno 111] Connection refused
jupyterhub  | 05:34:54.980 [ConfigProxy] e[32minfoe[39m: Proxying https://*:8080 to (no default)
jupyterhub  | 05:34:54.983 [ConfigProxy] e[32minfoe[39m: Proxy API at http://127.0.0.1:8001/api/routes
jupyterhub  | [D 2024-04-19 05:34:55.045 JupyterHub utils:274] Server at 127.0.0.1:8001 responded in 0.30s
jupyterhub  | [D 2024-04-19 05:34:55.167 JupyterHub utils:274] Server at 7735acaea79a:8080 responded in 0.42s
jupyterhub  | [D 2024-04-19 05:34:55.167 JupyterHub proxy:831] Proxy started and appears to be up
jupyterhub  | [D 2024-04-19 05:34:55.168 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 05:34:55.176 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
jupyterhub  | [I 2024-04-19 05:34:55.178 JupyterHub app:3204] Hub API listening on http://hub:8083/hub/
jupyterhub  | [D 2024-04-19 05:34:55.178 JupyterHub proxy:390] Fetching routes to check
jupyterhub  | [D 2024-04-19 05:34:55.178 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 05:34:55.179 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
jupyterhub  | [D 2024-04-19 05:34:55.180 JupyterHub proxy:393] Checking routes
jupyterhub  | [I 2024-04-19 05:34:55.180 JupyterHub proxy:478] Adding route for Hub: / => http://hub:8083
jupyterhub  | [D 2024-04-19 05:34:55.180 JupyterHub proxy:924] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
jupyterhub  | 05:34:55.182 [ConfigProxy] e[32minfoe[39m: Adding route / -> http://hub:8083
jupyterhub  | 05:34:55.183 [ConfigProxy] e[32minfoe[39m: Route added / -> http://hub:8083
jupyterhub  | 05:34:55.183 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/ 
jupyterhub  | [I 2024-04-19 05:34:55.184 JupyterHub app:3271] JupyterHub is now running at https://:8080/
jupyterhub  | [D 2024-04-19 05:34:55.185 JupyterHub app:2878] It took 0.626 seconds for the Hub to start
jupyterhub  | [D 2024-04-19 05:34:57.863 JupyterHub base:407] Refreshing auth for fleischer
jupyterhub  | [D 2024-04-19 05:34:57.867 JupyterHub user:431] Creating <class 'dockerspawner.dockerspawner.DockerSpawner'> for fleischer:
jupyterhub  | [W 2024-04-19 05:34:57.872 JupyterHub configurable:214] Config option `image_homedir_format_string` not recognized by `DockerSpawner`.
jupyterhub  | [W 2024-04-19 05:34:57.872 JupyterHub configurable:214] Config option `image_homedir_propagation` not recognized by `DockerSpawner`.
jupyterhub  | [D 2024-04-19 05:34:57.872 JupyterHub dockerspawner:960] Loaded state for fleischer: container name=jupyter-fleischer, id=544a0a2709984577cdcad4b1350266114de871d36f709cf335652345085661b9
jupyterhub  | [I 2024-04-19 05:34:57.904 JupyterHub log:192] 200 GET /hub/home (fleischer@::ffff:10.125.100.90) 59.26ms
jupyterhub  | [D 2024-04-19 05:34:58.076 JupyterHub log:192] 200 GET /hub/static/js/home.js?v=20240419053454 (@::ffff:10.125.100.90) 12.92ms
jupyterhub  | [D 2024-04-19 05:34:58.165 JupyterHub log:192] 200 GET /hub/static/components/moment/moment.js?v=20240419053454 (@::ffff:10.125.100.90) 2.28ms
jupyterhub  | [D 2024-04-19 05:34:58.207 JupyterHub log:192] 200 GET /hub/static/js/jhapi.js?v=20240419053454 (@::ffff:10.125.100.90) 2.01ms
jupyterhub  | [D 2024-04-19 05:34:58.262 JupyterHub log:192] 200 GET /hub/static/js/utils.js?v=20240419053454 (@::ffff:10.125.100.90) 1.83ms
jupyterhub  | [I 2024-04-19 05:35:53.131 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:192.168.125.23) 1.74ms
jupyterhub  | [I 2024-04-19 05:35:53.149 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:172.21.0.1) 1.51ms
jupyterhub  | [D 2024-04-19 05:35:53.162 JupyterHub handlers:36] Adding /usr/local/lib/python3.10/dist-packages/nativeauthenticator/templates to template path
jupyterhub  | [I 2024-04-19 05:35:53.163 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:FvTRwb8qeACB4R9zM03VxNjjvvxOH9Sr3bWePzE9V60=' {'path': '/hub/', 'max_age': 3600}
jupyterhub  | [I 2024-04-19 05:35:53.195 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:172.21.0.1) 33.71ms
jupyterhub  | [D 2024-04-19 05:35:56.958 JupyterHub scopes:884] Checking access to /hub/spawn/fleischer via scope servers
jupyterhub  | [D 2024-04-19 05:35:56.958 JupyterHub scopes:697] Argument-based access to /hub/spawn/fleischer via servers
jupyterhub  | [D 2024-04-19 05:35:56.959 JupyterHub pages:210] Serving options form for fleischer
jupyterhub  | [I 2024-04-19 05:35:56.972 JupyterHub log:192] 200 GET /hub/spawn/fleischer (fleischer@::ffff:10.125.100.90) 26.18ms
jupyterhub  | [D 2024-04-19 05:35:57.881 JupyterHub scopes:884] Checking access to /hub/spawn/fleischer via scope servers
jupyterhub  | [D 2024-04-19 05:35:57.882 JupyterHub scopes:697] Argument-based access to /hub/spawn/fleischer via servers
jupyterhub  | [D 2024-04-19 05:35:57.882 JupyterHub pages:258] Triggering spawn with supplied form options for fleischer
jupyterhub  | [D 2024-04-19 05:35:57.883 JupyterHub base:1061] Initiating spawn for fleischer
jupyterhub  | [D 2024-04-19 05:35:57.883 JupyterHub base:1065] 0/100 concurrent spawns
jupyterhub  | [D 2024-04-19 05:35:57.884 JupyterHub base:1070] 0 active servers
jupyterhub  | [I 2024-04-19 05:35:57.909 JupyterHub provider:660] Creating oauth client jupyterhub-user-fleischer
jupyterhub  | [D 2024-04-19 05:35:57.925 JupyterHub jupyterhub_config:1505] {'/home/ils_ubuntu/dockerhome/jupyterhub_stack_dockerspawner/jupyterconf/userconfs': {'bind': '/home/userconfs', 'mode': 'rw', 'propagation': 'rshared'}, '/etc/auto.jupyter.d': {'bind': '/etc/auto.jupyter.d', 'mode': 'rw'}, '/etc/auto.srv-fs1-box': {'bind': '/etc/auto.srv-fs1-box', 'mode': 'ro'}, '/etc/auto.srv-geo2': {'bind': '/etc/auto.srv-geo2', 'mode': 'ro'}, '/etc/auto.master.jupyter.d': {'bind': '/etc/auto.master.jupyter.d', 'mode': 'rw'}}
jupyterhub  | [D 2024-04-19 05:35:57.926 JupyterHub jupyterhub_config:1506] 1003
jupyterhub  | [D 2024-04-19 05:35:57.926 JupyterHub user:797] Calling Spawner.start for fleischer
jupyterhub  | [D 2024-04-19 05:35:57.950 JupyterHub dockerspawner:1027] Getting container 'jupyter-fleischer'
jupyterhub  | [I 2024-04-19 05:35:57.955 JupyterHub dockerspawner:1320] Found existing container jupyter-fleischer (id: 544a0a2)
jupyterhub  | [I 2024-04-19 05:35:57.956 JupyterHub dockerspawner:1335] Starting container jupyter-fleischer (id: 544a0a2)
jupyterhub  | [I 2024-04-19 05:35:58.345 JupyterHub provider:662] Updating oauth client jupyterhub-user-fleischer
jupyterhub  | [D 2024-04-19 05:35:58.351 JupyterHub spawner:1388] Polling subprocess every 30s
jupyterhub  | [D 2024-04-19 05:35:58.352 JupyterHub dockerspawner:972] Persisting state for fleischer: container name=jupyter-fleischer, id=544a0a2709984577cdcad4b1350266114de871d36f709cf335652345085661b9
jupyterhub  | [D 2024-04-19 05:35:58.353 JupyterHub utils:286] Waiting 2400s for server at http://172.21.0.3:8888/user/fleischer/
jupyterhub  | [I 2024-04-19 05:35:58.885 JupyterHub log:192] 302 POST /hub/spawn/fleischer?_xsrf=[secret] -> /hub/spawn-pending/fleischer?_xsrf=[secret] (fleischer@::ffff:10.125.100.90) 1009.07ms
jupyterhub  | [D 2024-04-19 05:35:58.913 JupyterHub scopes:884] Checking access to /hub/spawn-pending/fleischer via scope servers
jupyterhub  | [D 2024-04-19 05:35:58.913 JupyterHub scopes:697] Argument-based access to /hub/spawn-pending/fleischer via servers
jupyterhub  | [I 2024-04-19 05:35:58.914 JupyterHub pages:399] fleischer is pending spawn
jupyterhub  | [I 2024-04-19 05:35:58.920 JupyterHub log:192] 200 GET /hub/spawn-pending/fleischer?_xsrf=[secret] (fleischer@::ffff:10.125.100.90) 11.47ms
jupyterhub  | [D 2024-04-19 05:35:59.078 JupyterHub scopes:884] Checking access to /hub/api/users/fleischer/server/progress via scope read:servers
jupyterhub  | [D 2024-04-19 05:35:59.078 JupyterHub scopes:697] Argument-based access to /hub/api/users/fleischer/server/progress via read:servers
jupyterhub  | [I 2024-04-19 05:36:00.379 JupyterHub log:192] 200 GET /hub/api (@172.21.0.3) 2.33ms
jupyterhub  | [D 2024-04-19 05:36:00.434 JupyterHub scopes:884] Checking access to /hub/api/users/fleischer/activity via scope users:activity
jupyterhub  | [D 2024-04-19 05:36:00.434 JupyterHub scopes:697] Argument-based access to /hub/api/users/fleischer/activity via users:activity
jupyterhub  | [D 2024-04-19 05:36:00.436 JupyterHub users:882] Activity for user fleischer: 2024-04-19T05:36:00.355210Z
jupyterhub  | [D 2024-04-19 05:36:00.436 JupyterHub users:900] Activity on server fleischer/: 2024-04-19T05:36:00.355210Z
jupyterhub  | [I 2024-04-19 05:36:00.447 JupyterHub log:192] 200 POST /hub/api/users/fleischer/activity (fleischer@172.21.0.3) 25.71ms
jupyterhub  | [D 2024-04-19 05:36:07.886 JupyterHub dockerspawner:1027] Getting container 'jupyter-fleischer'
jupyterhub  | [D 2024-04-19 05:36:07.894 JupyterHub dockerspawner:1012] Container 544a0a2 status: {'Dead': False,
jupyterhub  |      'Error': '',
jupyterhub  |      'ExitCode': 0,
jupyterhub  |      'FinishedAt': '2024-04-19T05:34:13.2041519Z',
jupyterhub  |      'Health': {'FailingStreak': 0,
jupyterhub  |                 'Log': [{'End': '2024-04-19T07:34:06.4845099+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:34:06.1695631+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:34:09.8078331+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:34:09.4865026+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:34:13.1143984+02:00',
jupyterhub  |                          'ExitCode': 1,
jupyterhub  |                          'Output': 'Traceback (most recent call last):\n'
jupyterhub  |                                    '  File "/etc/jupyter/docker_healthcheck.py", '
jupyterhub  |                                    'line 27, in <module>\n'
jupyterhub  |                                    '    json_file = '
jupyterhub  |                                    'next(runtime_dir.glob("*server-*.json"))\n'
jupyterhub  |                                    '                '
jupyterhub  |                                    '^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
jupyterhub  |                                    'StopIteration\n',
jupyterhub  |                          'Start': '2024-04-19T07:34:12.8090904+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:01.6390004+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:01.3275372+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:04.9537091+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:04.6410549+02:00'}],
jupyterhub  |                 'Status': 'healthy'},
jupyterhub  |      'OOMKilled': False,
jupyterhub  |      'Paused': False,
jupyterhub  |      'Pid': 1072024,
jupyterhub  |      'Restarting': False,
jupyterhub  |      'Running': True,
jupyterhub  |      'StartedAt': '2024-04-19T05:35:58.321304Z',
jupyterhub  |      'Status': 'running'}
jupyterhub  | [W 2024-04-19 05:36:07.895 JupyterHub base:1222] User fleischer is slow to become responsive (timeout=10)
jupyterhub  | [D 2024-04-19 05:36:07.895 JupyterHub base:1227] Expecting server for fleischer at: http://172.21.0.3:8888/user/fleischer/
jupyterhub  | [D 2024-04-19 05:36:28.353 JupyterHub dockerspawner:1027] Getting container 'jupyter-fleischer'
jupyterhub  | [D 2024-04-19 05:36:28.360 JupyterHub dockerspawner:1012] Container 544a0a2 status: {'Dead': False,
jupyterhub  |      'Error': '',
jupyterhub  |      'ExitCode': 0,
jupyterhub  |      'FinishedAt': '2024-04-19T05:34:13.2041519Z',
jupyterhub  |      'Health': {'FailingStreak': 0,
jupyterhub  |                 'Log': [{'End': '2024-04-19T07:36:14.9071794+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:14.5895114+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:18.2159101+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:17.9083456+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:21.5404487+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:21.2177602+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:24.8531461+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:24.5422534+02:00'},
jupyterhub  |                         {'End': '2024-04-19T07:36:28.1665696+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T07:36:27.8545012+02:00'}],
jupyterhub  |                 'Status': 'healthy'},
jupyterhub  |      'OOMKilled': False,
jupyterhub  |      'Paused': False,
jupyterhub  |      'Pid': 1072024,
jupyterhub  |      'Restarting': False,
jupyterhub  |      'Running': True,
jupyterhub  |      'StartedAt': '2024-04-19T05:35:58.321304Z',
jupyterhub  |      'Status': 'running'}
jupyterhub  | [D 2024-04-19 05:36:58.353 JupyterHub dockerspawner:1027] Getting container 'jupyter-fleischer'
[... repeats itself for 2400 seconds - like set timeout - ]
jupyterhub  | [W 2024-04-19 06:18:12.695 JupyterHub user:936] fleischer's server never showed up at http://172.21.0.3:8888/user/fleischer/ after 2400 seconds. Giving up.
jupyterhub  |     
jupyterhub  |     Common causes of this timeout, and debugging tips:
jupyterhub  |     
jupyterhub  |     1. The server didn't finish starting,
jupyterhub  |        or it crashed due to a configuration issue.
jupyterhub  |        Check the single-user server's logs for hints at what needs fixing.
jupyterhub  |     2. The server started, but is not accessible at the specified URL.
jupyterhub  |        This may be a configuration issue specific to your chosen Spawner.
jupyterhub  |        Check the single-user server logs and resource to make sure the URL
jupyterhub  |        is correct and accessible from the Hub.
jupyterhub  |     3. (unlikely) Everything is working, but the server took too long to respond.
jupyterhub  |        To fix: increase `Spawner.http_timeout` configuration
jupyterhub  |        to a number of seconds that is enough for servers to become responsive.
jupyterhub  |     
jupyterhub  | [D 2024-04-19 06:18:12.695 JupyterHub user:985] Stopping fleischer
jupyterhub  | [D 2024-04-19 06:18:12.696 JupyterHub dockerspawner:1027] Getting container 'jupyter-fleischer'
jupyterhub  | [D 2024-04-19 06:18:12.703 JupyterHub dockerspawner:1012] Container 544a0a2 status: {'Dead': False,
jupyterhub  |      'Error': '',
jupyterhub  |      'ExitCode': 0,
jupyterhub  |      'FinishedAt': '2024-04-19T05:34:13.2041519Z',
jupyterhub  |      'Health': {'FailingStreak': 0,
jupyterhub  |                 'Log': [{'End': '2024-04-19T08:17:58.3957909+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T08:17:58.0840256+02:00'},
jupyterhub  |                         {'End': '2024-04-19T08:18:01.7243605+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T08:18:01.397396+02:00'},
jupyterhub  |                         {'End': '2024-04-19T08:18:05.0391487+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T08:18:04.7258328+02:00'},
jupyterhub  |                         {'End': '2024-04-19T08:18:08.3613513+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T08:18:08.0411762+02:00'},
jupyterhub  |                         {'End': '2024-04-19T08:18:11.6742217+02:00',
jupyterhub  |                          'ExitCode': 0,
jupyterhub  |                          'Output': 'b\'{"version": "2.13.0"}\'\n',
jupyterhub  |                          'Start': '2024-04-19T08:18:11.3622631+02:00'}],
jupyterhub  |                 'Status': 'healthy'},
jupyterhub  |      'OOMKilled': False,
jupyterhub  |      'Paused': False,
jupyterhub  |      'Pid': 1072024,
jupyterhub  |      'Restarting': False,
jupyterhub  |      'Running': True,
jupyterhub  |      'StartedAt': '2024-04-19T05:35:58.321304Z',
jupyterhub  |      'Status': 'running'}
jupyterhub  | [I 2024-04-19 06:18:12.703 JupyterHub dockerspawner:1424] Stopping container jupyter-fleischer (id: 544a0a2)
jupyterhub  | [D 2024-04-19 06:18:13.242 JupyterHub user:1008] Finished stopping fleischer
jupyterhub  | [D 2024-04-19 06:18:13.247 JupyterHub dockerspawner:972] Persisting state for fleischer: container name=jupyter-fleischer, id=544a0a2709984577cdcad4b1350266114de871d36f709cf335652345085661b9
jupyterhub  | [E 2024-04-19 06:18:13.248 JupyterHub gen:630] Exception in Future <Task finished name='Task-99' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/base.py:1081> exception=TimeoutError("Server at http://172.21.0.3:8888/user/fleischer/ didn't respond in 2400 seconds")> after timeout
jupyterhub  |     Traceback (most recent call last):
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 625, in error_callback
jupyterhub  |         future.result()
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/base.py", line 1088, in finish_user_spawn
jupyterhub  |         await spawn_future
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 917, in spawn
jupyterhub  |         await self._wait_up(spawner)
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 961, in _wait_up
jupyterhub  |         raise e
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 931, in _wait_up
jupyterhub  |         resp = await server.wait_up(
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/utils.py", line 316, in wait_for_http_server
jupyterhub  |         re = await exponential_backoff(
jupyterhub  |       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/utils.py", line 257, in exponential_backoff
jupyterhub  |         raise asyncio.TimeoutError(fail_message)
jupyterhub  |     asyncio.exceptions.TimeoutError: Server at http://172.21.0.3:8888/user/fleischer/ didn't respond in 2400 seconds
jupyterhub  |     
jupyterhub  | [W 2024-04-19 06:18:13.250 JupyterHub users:674] Stream closed while handling /hub/api/users/fleischer/server/progress?_xsrf=MnwxOjB8MTA6MTcxMjkyODc5NXw1Ol94c3JmfDg4OllUaGpOemRpWWpsak4yVmtOR014TkRoak5tTTRaalF4TnpkaE1EUXhaRFE2TVdZMVlUazJNbVJtTjJKaE5HRTVaR0ZqWkRoa09USXdNbVF6TXprMFltVT18NTQ3NGY5OTBiOTYyNmI5ZjM1NmJjZjY4MzdkNDRiMDFiMGI5YTI0Yjg0MWZhYjllNjNjOWVjYzY0MzVlOGFkNg
jupyterhub  | [I 2024-04-19 06:18:13.250 JupyterHub log:192] 200 GET /hub/api/users/fleischer/server/progress?_xsrf=[secret] (fleischer@::ffff:10.125.100.90) 2534176.19ms
jupyterhub  | [D 2024-04-19 06:19:55.184 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 06:19:55.188 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
jupyterhub  | [D 2024-04-19 06:19:55.189 JupyterHub proxy:393] Checking routes
jupyterhub  | [I 2024-04-19 06:20:53.352 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:172.21.0.1) 1.66ms
jupyterhub  | [I 2024-04-19 06:20:53.364 JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:172.21.0.1) 1.48ms
jupyterhub  | [I 2024-04-19 06:20:53.374 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:FvTRwb8qeACB4R9zM03VxNjjvvxOH9Sr3bWePzE9V60=' {'path': '/hub/', 'max_age': 3600}
jupyterhub  | [I 2024-04-19 06:20:53.376 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:172.21.0.1) 3.33ms
jupyterhub  | [D 2024-04-19 06:24:55.184 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 06:24:55.187 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
jupyterhub  | [D 2024-04-19 06:24:55.188 JupyterHub proxy:393] Checking routes
jupyterhub  | [D 2024-04-19 06:29:55.185 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 06:29:55.188 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes 
jupyterhub  | [D 2024-04-19 06:29:55.189 JupyterHub proxy:393] Checking routes
jupyterhub  | [I 2024-04-19 06:34:40.838 JupyterHub metrics:279] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
jupyterhub  | [I 2024-04-19 06:34:40.840 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.seven_days
jupyterhub  | [I 2024-04-19 06:34:40.842 JupyterHub metrics:279] Found 2 active users in the last ActiveUserPeriods.thirty_days
jupyterhub  | [D 2024-04-19 06:34:54.694 JupyterHub app:2364] Purging expired APITokens
jupyterhub  | [D 2024-04-19 06:34:54.696 JupyterHub app:2364] Purging expired OAuthCodes
jupyterhub  | [D 2024-04-19 06:34:55.184 JupyterHub proxy:924] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
jupyterhub  | 06:34:55.187 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes

jannefleischer · April 19, 2024, 9:45am

And this is Part 1 of the log from my spawned container jupyter-fleischer:

Entered start.sh with args: jupyterhub-singleuser --debug
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/start-notebook.d
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1000 gid: 100
Sourcing shell script: /usr/local/bin/before-notebook.d/10activate-conda-env.sh
Done running hooks in: /usr/local/bin/before-notebook.d
Executing the command: jupyterhub-singleuser --debug
[D 2024-04-19 05:30:32.020 ServerApp] Searching ['/home/userconfs/fleischer', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2024-04-19 05:30:32.020 ServerApp] Looking for jupyter_config in /etc/jupyter
[D 2024-04-19 05:30:32.021 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-04-19 05:30:32.021 ServerApp] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2024-04-19 05:30:32.021 ServerApp] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2024-04-19 05:30:32.021 ServerApp] Looking for jupyter_config in /home/userconfs/fleischer
[D 2024-04-19 05:30:32.022 ServerApp] Looking for jupyter_server_config in /etc/jupyter
[D 2024-04-19 05:30:32.022 ServerApp] Loaded config file: /etc/jupyter/jupyter_server_config.py
[D 2024-04-19 05:30:32.023 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-04-19 05:30:32.023 ServerApp] Looking for jupyter_server_config in /opt/conda/etc/jupyter
[D 2024-04-19 05:30:32.023 ServerApp] Looking for jupyter_server_config in /home/jovyan/.local/etc/jupyter
[D 2024-04-19 05:30:32.023 ServerApp] Looking for jupyter_server_config in /home/userconfs/fleischer
[D 2024-04-19 05:30:32.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:32.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:32.027 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:32.027 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:32.027 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/userconfs/fleischer/jupyter_server_config.json
[D 2024-04-19 05:30:32.032 ServerApp] Extension package jupyterhub took 0.0000s to import
[D 2024-04-19 05:30:32.043 ServerApp] Extension package jupyter_lsp took 0.0103s to import
[D 2024-04-19 05:30:32.048 ServerApp] Extension package jupyter_server_terminals took 0.0044s to import
[D 2024-04-19 05:30:32.109 ServerApp] Extension package jupyterlab took 0.0605s to import
[D 2024-04-19 05:30:32.692 ServerApp] Paths used for configuration of page_config: 
    	/opt/conda/etc/jupyter/labconfig/page_config.json
[D 2024-04-19 05:30:32.693 ServerApp] Paths used for configuration of page_config: 
    	
[D 2024-04-19 05:30:32.708 ServerApp] Extension package nbclassic took 0.0034s to import
[W 2024-04-19 05:30:32.710 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[D 2024-04-19 05:30:32.710 ServerApp] Extension package notebook took 0.0000s to import
[D 2024-04-19 05:30:32.713 ServerApp] Extension package notebook_shim took 0.0000s to import
[I 2024-04-19 05:30:32.713 ServerApp] jupyter_lsp | extension was successfully linked.
[D 2024-04-19 05:30:32.717 TerminalsExtensionApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue {'update': {'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[D 2024-04-19 05:30:32.718 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-04-19 05:30:32.719 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-04-19 05:30:32.719 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.4
[I 2024-04-19 05:30:32.720 JupyterHubSingleUser] Using default url from server extension lab: /lab
[D 2024-04-19 05:30:32.720 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[D 2024-04-19 05:30:32.724 JupyterHubSingleUser] Config changed: {'ServerApp': {'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'ip': '127.0.0.1', 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': '', 'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[D 2024-04-19 05:30:32.725 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-04-19 05:30:32.726 ServerApp] jupyterhub | extension was successfully linked.
[W 2024-04-19 05:30:32.728 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[D 2024-04-19 05:30:32.731 LabApp] Config changed: {'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']}>}, 'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-04-19 05:30:32.732 ServerApp] jupyterlab | extension was successfully linked.
[W 2024-04-19 05:30:32.734 NotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[D 2024-04-19 05:30:32.737 NotebookApp] Config changed: {'NotebookApp': {'extra_template_paths': <LazyConfigValue value=['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']>}, 'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-04-19 05:30:32.737 ServerApp] nbclassic | extension was successfully linked.
[W 2024-04-19 05:30:32.739 JupyterNotebookApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[D 2024-04-19 05:30:32.742 JupyterNotebookApp] Config changed: {'NotebookApp': {'extra_template_paths': <LazyConfigValue value=['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']>}, 'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-04-19 05:30:32.743 ServerApp] notebook | extension was successfully linked.
[I 2024-04-19 05:30:32.744 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[D 2024-04-19 05:30:32.991 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/home/userconfs/fleischer/jupyter_notebook_config.json
[D 2024-04-19 05:30:32.991 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:32.991 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:32.992 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:32.992 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:32.992 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/home/userconfs/fleischer/jupyter_notebook_config.json
[I 2024-04-19 05:30:32.993 ServerApp] notebook_shim | extension was successfully linked.
[D 2024-04-19 05:30:32.994 ServerApp] Config changed: {'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue value=['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']>}, 'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}}
/opt/conda/lib/python3.11/site-packages/jupyter_server/serverapp.py:2235: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/fleischer/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/fleischer/oauth_callback
  self.web_app = ServerWebApplication(
[I 2024-04-19 05:30:33.013 ServerApp] notebook_shim | extension was successfully loaded.
[D 2024-04-19 05:30:33.014 ServerApp] [lsp] rootUri will be file:///home/jovyan
[D 2024-04-19 05:30:33.015 ServerApp] [lsp] virtualDocumentsUri will be file:///home/jovyan/.virtual_documents
[I 2024-04-19 05:30:33.015 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-04-19 05:30:33.016 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[D 2024-04-19 05:30:33.020 ServerApp] Paths used for configuration of jupyter_config: 
    	/etc/jupyter/jupyter_config.json
[D 2024-04-19 05:30:33.020 ServerApp] Paths used for configuration of jupyter_config: 
    	/usr/local/etc/jupyter/jupyter_config.json
[D 2024-04-19 05:30:33.021 ServerApp] Paths used for configuration of jupyter_config: 
    	/opt/conda/etc/jupyter/jupyter_config.json
[D 2024-04-19 05:30:33.021 ServerApp] Paths used for configuration of jupyter_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_config.json
[D 2024-04-19 05:30:33.022 ServerApp] Paths used for configuration of jupyter_config: 
    	/home/userconfs/fleischer/jupyter_config.json
[D 2024-04-19 05:30:33.022 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:33.023 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:33.024 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:33.025 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_notebook_config.json
[D 2024-04-19 05:30:33.026 ServerApp] Paths used for configuration of jupyter_notebook_config: 
    	/home/userconfs/fleischer/jupyter_notebook_config.json
[D 2024-04-19 05:30:33.026 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:33.027 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:33.027 JupyterHubSingleUser] jupyterhub and jupyterhub-singleuser both on version 4.1.4
[I 2024-04-19 05:30:33.028 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-04-19 05:30:33.028 ServerApp] jupyterhub | extension was successfully loaded.
[D 2024-04-19 05:30:33.030 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter-lsp-jupyter-server.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook.json
    	/opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    	/opt/conda/etc/jupyter/jupyter_server_config.json
[I 2024-04-19 05:30:33.031 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2024-04-19 05:30:33.031 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2024-04-19 05:30:33.031 LabApp] Extension Manager is 'pypi'.
[D 2024-04-19 05:30:33.032 LabApp] Plugins in PyPIExtensionManager will managed on the sys_prefix level
[D 2024-04-19 05:30:33.033 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/jovyan/.local/etc/jupyter/jupyter_server_config.json
[D 2024-04-19 05:30:33.033 ServerApp] Paths used for configuration of jupyter_server_config: 
    	/home/userconfs/fleischer/jupyter_server_config.json
[D 2024-04-19 05:30:33.043 LabApp] Extensions list will be fetched from https://pypi.org/pypi.
[D 2024-04-19 05:30:33.044 LabApp] Plugins in PluginManager will managed on the sys_prefix level
[I 2024-04-19 05:30:33.047 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-04-19 05:30:33.052 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-04-19 05:30:33.054 ServerApp] notebook | extension was successfully loaded.
[I 2024-04-19 05:30:33.055 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2024-04-19 05:30:33.055 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-04-19 05:30:33.055 ServerApp] http://127.0.0.1:8888/user/fleischer/lab?token=...
[I 2024-04-19 05:30:33.055 ServerApp]     http://127.0.0.1:8888/user/fleischer/lab?token=...
[I 2024-04-19 05:30:33.055 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 2024-04-19 05:30:33.057 JupyterHubSingleUser] Notifying Hub of activity 2024-04-19T05:30:33.003005Z
[D 2024-04-19 05:30:33.500 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.501 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.501 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.501 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.501 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.502 ServerApp] bash-language-server/out/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.502 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.503 ServerApp] bash-language-server/bin/main.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /home/jovyan/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.503 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.504 ServerApp] dockerfile-language-server-nodejs/lib/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /home/jovyan/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.504 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.505 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.505 ServerApp] javascript-typescript-langserver/lib/language-server-stdio.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /home/jovyan/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.506 ServerApp] pyright/langserver.index.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.507 ServerApp] Checking for /home/jovyan/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.508 ServerApp] sql-language-server/dist/bin/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.508 ServerApp] Checking for /home/jovyan/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.509 ServerApp] typescript-language-server/lib/cli.mjs not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /home/jovyan/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.509 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.510 ServerApp] unified-language-server/src/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /home/jovyan/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] vscode-css-languageserver-bin/cssServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /home/jovyan/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.510 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] vscode-html-languageserver-bin/htmlServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /home/jovyan/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.511 ServerApp] vscode-json-languageserver-bin/jsonServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /home/jovyan/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.511 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.512 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.512 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.512 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.512 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.512 ServerApp] yaml-language-server/bin/yaml-language-server not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]


[...]

jannefleischer · April 19, 2024, 9:45am

And this is Part 2 of the log from my spawned container jupyter-fleischer:

[I 2024-04-19 05:30:33.512 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.521 ServerApp] bash-language-server/out/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] bash-language-server/bin/main.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/out/cli.js
[D 2024-04-19 05:30:33.522 ServerApp] bash-language-server/out/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /home/jovyan/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/lib/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] Checking for /opt/conda/node_modules/bash-language-server/bin/main.js
[D 2024-04-19 05:30:33.522 ServerApp] bash-language-server/bin/main.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /home/jovyan/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.523 ServerApp] dockerfile-language-server-nodejs/lib/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /home/jovyan/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/lib/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/node_modules/dockerfile-language-server-nodejs/lib/server.js
[D 2024-04-19 05:30:33.524 ServerApp] dockerfile-language-server-nodejs/lib/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /home/jovyan/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.524 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] javascript-typescript-langserver/lib/language-server-stdio.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /home/jovyan/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/lib/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] Checking for /opt/conda/node_modules/javascript-typescript-langserver/lib/language-server-stdio.js
[D 2024-04-19 05:30:33.525 ServerApp] javascript-typescript-langserver/lib/language-server-stdio.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /home/jovyan/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] pyright/langserver.index.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /home/jovyan/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/lib/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] Checking for /opt/conda/node_modules/pyright/langserver.index.js
[D 2024-04-19 05:30:33.527 ServerApp] pyright/langserver.index.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /home/jovyan/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] sql-language-server/dist/bin/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /home/jovyan/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/lib/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] Checking for /opt/conda/node_modules/sql-language-server/dist/bin/cli.js
[D 2024-04-19 05:30:33.531 ServerApp] sql-language-server/dist/bin/cli.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.532 ServerApp] Checking for /home/jovyan/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.532 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] typescript-language-server/lib/cli.mjs not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /home/jovyan/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/lib/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] Checking for /opt/conda/node_modules/typescript-language-server/lib/cli.mjs
[D 2024-04-19 05:30:33.533 ServerApp] typescript-language-server/lib/cli.mjs not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /home/jovyan/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] unified-language-server/src/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /home/jovyan/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/lib/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] Checking for /opt/conda/node_modules/unified-language-server/src/server.js
[D 2024-04-19 05:30:33.534 ServerApp] unified-language-server/src/server.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /home/jovyan/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] vscode-css-languageserver-bin/cssServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /home/jovyan/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] Checking for /opt/conda/node_modules/vscode-css-languageserver-bin/cssServerMain.js
[D 2024-04-19 05:30:33.535 ServerApp] vscode-css-languageserver-bin/cssServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /home/jovyan/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] vscode-html-languageserver-bin/htmlServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /home/jovyan/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] Checking for /opt/conda/node_modules/vscode-html-languageserver-bin/htmlServerMain.js
[D 2024-04-19 05:30:33.536 ServerApp] vscode-html-languageserver-bin/htmlServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /home/jovyan/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] vscode-json-languageserver-bin/jsonServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /home/jovyan/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/lib/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.537 ServerApp] Checking for /opt/conda/node_modules/vscode-json-languageserver-bin/jsonServerMain.js
[D 2024-04-19 05:30:33.538 ServerApp] vscode-json-languageserver-bin/jsonServerMain.js not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /home/jovyan/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] yaml-language-server/bin/yaml-language-server not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /home/jovyan/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/share/jupyter/lab/staging/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.538 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.539 ServerApp] Checking for /opt/conda/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.539 ServerApp] Checking for /opt/conda/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-04-19 05:30:33.539 ServerApp] yaml-language-server/bin/yaml-language-server not found in node_modules of [PosixPath('/home/jovyan'), PosixPath('/opt/conda/share/jupyter/lab/staging'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda'), PosixPath('/opt/conda/lib'), PosixPath('/opt/conda')]
[D 2024-04-19 05:30:33.559 ServerApp] [lsp] None of the installed servers require virtual documents disabling shadow filesystem.
[D 2024-04-19 05:30:33.559 ServerApp] [lsp] The following Language Servers will be available: {}
[D 2024-04-19 05:30:34.232 ServerApp] No user identified
[I 2024-04-19 05:30:34.232 ServerApp] 200 GET /user/fleischer/api (@127.0.0.1) 1.64ms
[D 2024-04-19 05:30:37.548 ServerApp] No user identified
[I 2024-04-19 05:30:37.549 ServerApp] 200 GET /user/fleischer/api (@127.0.0.1) 1.84ms
[D 2024-04-19 05:30:40.869 ServerApp] No user identified
[I 2024-04-19 05:30:40.870 ServerApp] 200 GET /user/fleischer/api (@127.0.0.1) 1.88ms
[D 2024-04-19 05:30:44.188 ServerApp] No user identified
[I 2024-04-19 05:30:44.188 ServerApp] 200 GET /user/fleischer/api (@127.0.0.1) 1.84ms
[D 2024-04-19 05:30:47.519 ServerApp] No user identified
[... repeats itseelf for tiemout set 2400 seconds ... ]
[C 2024-04-19 05:34:12.869 ServerApp] received signal 15, stopping
[I 2024-04-19 05:34:12.870 ServerApp] Shutting down 7 extensions
[D 2024-04-19 05:34:12.871 ServerApp] jupyter_server_terminals | extension app 'jupyter_server_terminals' stopping
[D 2024-04-19 05:34:12.871 ServerApp] jupyter_server_terminals | extension app 'jupyter_server_terminals' stopped
[D 2024-04-19 05:34:12.871 ServerApp] jupyterhub | extension app 'jupyterhub-singleuser' stopping
[D 2024-04-19 05:34:12.871 ServerApp] jupyterhub | extension app 'jupyterhub-singleuser' stopped
[D 2024-04-19 05:34:12.871 ServerApp] jupyterlab | extension app 'lab' stopping
[D 2024-04-19 05:34:12.871 ServerApp] jupyterlab | extension app 'lab' stopped
[D 2024-04-19 05:34:12.871 ServerApp] nbclassic | extension app 'nbclassic' stopping
[D 2024-04-19 05:34:12.871 ServerApp] nbclassic | extension app 'nbclassic' stopped
[D 2024-04-19 05:34:12.871 ServerApp] notebook | extension app 'notebook' stopping
[D 2024-04-19 05:34:12.871 ServerApp] notebook | extension app 'notebook' stopped

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|
                                                                           
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

mahendrapaipuri · April 19, 2024, 12:12pm

jannefleischer:

[D 2024-04-19 05:30:32.720 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '0.0.0.0', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}
[D 2024-04-19 05:30:32.724 JupyterHubSingleUser] Config changed: {'ServerApp': {'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'ip': '127.0.0.1', 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': '', 'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/templates']}>}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}}
[D 2024-04-19 05:30:32.725 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'ip': '127.0.0.1', 'open_browser': False, 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_lsp': True, 'jupyter_server_terminals': True, 'jupyterlab': True, 'nbclassic': True, 'notebook': True, 'notebook_shim': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 8888, 'base_url': '/user/fleischer/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'InlineBackend': {'figure_formats': {'pdf', 'svg', 'jpeg', 'png'}}, 'FileContentsManager': {'delete_to_trash': False}}

Here are some clues. Looks like your single user server is picking up some stale config from one of your config directories. Basically, the ip of single user server is being changed from all interfaces (0.0.0.0) to localhost (127.0.0.1) so I guess Hub is not able to reach the single servers.

Check the list of config files your single user servers are reading and remove those stale config params.

jannefleischer · April 22, 2024, 9:25am

This was the right hint! There where two Issues at hand with my config.

The first one was, that I ran jupyterlab without a config which defaults
c.ServerApp.allow_remote_access to False. (this is irrelevant for jupyterhub-singleuser)

The second, more important issue was, that I ignored a crucial part from the help-text in the config file:

...
#  Subclasses which launch remotely or in containers should override the default
#  to '0.0.0.0'.
#  
#  .. versionchanged:: 2.0
#      Default changed to '127.0.0.1', from ''.
#      In most cases, this does not result in a change in behavior,
#      as '' was interpreted as 'unspecified',
#      which used the subprocesses' own default, itself usually '127.0.0.1'.
#  Default: '127.0.0.1'
c.Spawner.ip = '127.0.0.1'

I just followed the …versionchanged-Part, blindly setting it to localhost. Instead I had to set it correctly to: c.Spawner.ip = '0.0.0.0'

Topic		Replies	Views
JupyterHub with Podman and DockerSpawner: Immediate 'JupyterLab Server Unavailable' Error on Login, Resolved by Refresh JupyterHub jupyterhub , help-wanted	8	807	February 1, 2024
Need some help with spawners JupyterHub jupyterhub , how-to , help-wanted	11	1568	July 10, 2022
Setting up JupyterHub and JupyterLab in docker JupyterHub jupyterlab , jupyterhub , docker	2	956	July 11, 2024
JupyterHub with DockerSpawner: Failed to connect to Hub API JupyterHub help-wanted	5	3385	March 25, 2021
JupyterHub DockerSpawner not preserving API token when attempting to spawn container with NB_USER JupyterHub help-wanted , docker	2	1499	February 28, 2022

Switch to Hub/Lab4: Jupyterhub and spawner Jupyterlab container can't communicate properly: "No user identified"

Related topics