of course: here’s the debug logs of a restart of the service with the reproduced error:
Mar 12 12:20:34 tljh systemd[1]: Stopping jupyterhub.service...
Mar 12 12:20:34 tljh python3[89773]: [C 2024-03-12 12:20:34.356 JupyterHub app:3336] Received signal SIGTERM, initiating shutdown...
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.356 JupyterHub app:2976] Cleaning up 1 services...
Mar 12 12:20:34 tljh python3[89773]: [D 2024-03-12 12:20:34.356 JupyterHub service:436] Stopping service cull-idle
Mar 12 12:20:34 tljh python3[89773]: [D 2024-03-12 12:20:34.356 JupyterHub spawner:1777] Interrupting 89777
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:2988] Leaving single-user servers running
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:2996] I didn't start the proxy, I can't clean it up
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:3013] ...done
Mar 12 12:20:34 tljh systemd[1]: jupyterhub.service: Deactivated successfully.
Mar 12 12:20:34 tljh systemd[1]: Stopped jupyterhub.service.
Mar 12 12:20:34 tljh systemd[1]: jupyterhub.service: Consumed 1.171s CPU time.
Mar 12 12:20:34 tljh systemd[1]: Started jupyterhub.service.
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2859] Running JupyterHub version 4.0.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.0.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.401 JupyterHub app:1664] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.401 JupyterHub app:1833] Connecting to db: sqlite:///jupyterhub.sqlite
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.412 JupyterHub orm:1016] database schema version found: 0eee8c825d24
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.417 JupyterHub orm:1016] database schema version found: 0eee8c825d24
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.454 JupyterHub app:2104] Loading roles into database
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.479 JupyterHub app:2343] Purging expired APITokens
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.480 JupyterHub app:2343] Purging expired OAuthCodes
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.482 JupyterHub app:2179] Loading role assignments from config
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.482 JupyterHub app:2250] Assigning 2 allowed_users to the user role
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.494 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.494 JupyterHub proxy:510] Setting up traefik's dynamic config...
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.506 JupyterHub fileprovider:95] Writing dynamic config {'http': {'routers': {'route_api': {'rule': 'Host(`localhost`) && PathPrefix(`/api`)', 'entryPoints': ['auth_api'], 'service': 'api@internal', 'middlewares': ['auth_api']}, 'router__2F': {'service': 'service__2F', 'rule': 'PathPrefix(`/`)', 'entryPoints': ['http']}}, 'middlewares': {'auth_api': {'basicAuth': {'users': ['api_admin:$apr1$GoQjK9yA$vJiIBIoiXJfM9QUxUbxuZ.']}}}, 'services': {'service__2F': {'loadBalancer': {'passHostHeader': True, 'servers': [{'url': 'http://127.0.0.1:15001'}]}}}}, 'jupyterhub': {'routes': {'router__2F': {'routespec': '/', 'target': 'http://127.0.0.1:15001', 'router': 'router__2F', 'service': 'service__2F', 'data': {'hub': True}}}}}
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.506 JupyterHub app:2502] Initializing spawners
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.508 JupyterHub app:2633] Loaded users:
Mar 12 12:20:35 tljh python3[89783]:
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.508 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/overview
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.509 JupyterHub app:2928] Initialized 0 spawners in 0.015 seconds
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.514 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.514 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.seven_days
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.515 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.thirty_days
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.515 JupyterHub app:3142] Not starting proxy
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.516 JupyterHub proxy:372] 200 GET http://localhost:8099/api/overview
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.516 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/entrypoints/http
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.517 JupyterHub proxy:372] 200 GET http://localhost:8099/api/entrypoints/http
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub app:3178] Hub API listening on http://127.0.0.1:15001/hub/
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub app:3189] Starting managed service cull-idle
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub service:385] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.518 JupyterHub service:133] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub spawner:1384] Polling subprocess every 30s
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub proxy:389] Fetching routes to check
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub proxy:392] Checking routes
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.519 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://127.0.0.1:15001/hub/
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub app:2852] It took 0.188 seconds for the Hub to start
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.610 JupyterHub base:299] Recording first activity for <APIToken('c378...', service='cull-idle', client_id='jupyterhub')>
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.615 JupyterHub log:191] 200 GET /hub/api/ (cull-idle@127.0.0.1) 6.44ms
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.617 JupyterHub scopes:877] Checking access to /hub/api/users via scope list:users
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.617 JupyterHub scopes:677] Unrestricted access to /hub/api/users via list:users
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.622 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 6.07ms
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.021 JupyterHub ldapauthenticator:256] Looking up user with:
Mar 12 12:20:38 tljh python3[89783]: search_base = 'ou=users,ou=domain,dc=domain,dc=local'
Mar 12 12:20:38 tljh python3[89783]: search_filter = '(sAMAccountName=gabbr)'
Mar 12 12:20:38 tljh python3[89783]: attributes = 'sAMAccountName'
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.023 JupyterHub ldapauthenticator:379] Attempting to bind Firstname Lastname with Firstname Lastname
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.148 JupyterHub ldapauthenticator:392] Status of user bind Firstname Lastname with Firstname Lastname : True
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.151 JupyterHub roles:281] Assigning default role to User gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:587] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:591] Setting cookie for gabbr: jupyterhub-hub-login
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.154 JupyterHub base:837] User logged in: gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub user:431] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for gabbr:
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.155 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (gabbr@192.168.169.1) 289.62ms
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.208 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.208 JupyterHub scopes:677] Unrestricted access to /hub/spawn via servers
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub pages:217] Triggering spawn with default options for gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:961] Initiating spawn for gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:965] 0/100 concurrent spawns
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:970] 0 active servers
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.228 JupyterHub provider:659] Creating oauth client jupyterhub-user-gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.239 JupyterHub user:794] Calling Spawner.start for gabbr
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.241 JupyterHub spawner:1689] Spawning jupyterhub-singleuser
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.253 JupyterHub spawner:1384] Polling subprocess every 30s
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.899 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.900 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.903 ServerApp] jupyterhub | extension was successfully linked.
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.923 JupyterHub log:191] 200 GET /hub/api (@127.0.0.1) 0.77ms
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.923 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.924 ServerApp] jupyterhub | extension was successfully loaded.
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Serving notebooks from local directory: /home/gabbr/jupyter
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Jupyter Server 2.12.5 is running at:
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] http://127.0.0.1:42543/user/gabbr/lab?token=...
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] http://127.0.0.1:42543/user/gabbr/lab?token=...
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.935 JupyterHub base:299] Recording first activity for <APIToken('c200...', user='gabbr', client_id='jupyterhub')>
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.940 JupyterHub scopes:877] Checking access to /hub/api/users/gabbr/activity via scope users:activity
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.940 JupyterHub scopes:690] Argument-based access to /hub/api/users/gabbr/activity via users:activity
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.941 JupyterHub users:874] Activity for user gabbr: 2024-03-12T12:20:38.911270Z
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.941 JupyterHub users:892] Activity on server gabbr/: 2024-03-12T12:20:38.911270Z
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.945 JupyterHub log:191] 200 POST /hub/api/users/gabbr/activity (gabbr@127.0.0.1) 11.94ms
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.211 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/gabbr (gabbr@192.168.169.1) 1013.18ms
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.266 JupyterHub scopes:877] Checking access to /hub/spawn-pending/gabbr via scope servers
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.266 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/gabbr via servers
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.266 JupyterHub pages:398] gabbr is pending spawn
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.287 JupyterHub log:191] 200 GET /hub/spawn-pending/gabbr (gabbr@192.168.169.1) 23.20ms
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.489 JupyterHub scopes:877] Checking access to /hub/api/users/gabbr/server/progress via scope read:servers
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.489 JupyterHub scopes:677] Unrestricted access to /hub/api/users/gabbr/server/progress via read:servers
Mar 12 12:20:40 tljh python3[89790]: [I 2024-03-12 12:20:40.252 ServerApp] 302 GET /user/gabbr/ -> /user/gabbr/lab? (@127.0.0.1) 0.43ms
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.252 JupyterHub utils:278] Server at http://127.0.0.1:42543/user/gabbr/ responded with 302
Mar 12 12:20:40 tljh python3[89783]: [W 2024-03-12 12:20:40.252 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.252 JupyterHub base:990] User gabbr took 2.044 seconds to start
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.253 JupyterHub proxy:330] Adding user gabbr to proxy /user/gabbr/ => http://127.0.0.1:42543
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.253 JupyterHub fileprovider:95] Writing dynamic config {'http': {'routers': {'route_api': {'rule': 'Host(`localhost`) && PathPrefix(`/api`)', 'entryPoints': ['auth_api'], 'service': 'api@internal', 'middlewares': ['auth_api']}, 'router__2F': {'service': 'service__2F', 'rule': 'PathPrefix(`/`)', 'entryPoints': ['http']}, 'router__2Fuser_2Fgabbr_2F': {'service': 'service__2Fuser_2Fgabbr_2F', 'rule': '( PathPrefix(`/user/gabbr/`) || Path(`/user/gabbr`) )', 'entryPoints': ['http']}}, 'middlewares': {'auth_api': {'basicAuth': {'users': ['api_admin:$apr1$GoQjK9yA$vJiIBIoiXJfM9QUxUbxuZ.']}}}, 'services': {'service__2F': {'loadBalancer': {'passHostHeader': True, 'servers': [{'url': 'http://127.0.0.1:15001'}]}}, 'service__2Fuser_2Fgabbr_2F': {'loadBalancer': {'servers': [{'url': 'http://127.0.0.1:42543'}], 'passHostHeader': True}}}}, 'jupyterhub': {'routes': {'router__2F': {'routespec': '/', 'target': 'http://127.0.0.1:15001', 'router': 'router__2F', 'service': 'service__2F', 'data': {'hub': True}}, 'router__2Fuser_2Fgabbr_2F': {'data': {'user': 'gabbr', 'server_name': ''}, 'routespec': '/user/gabbr/', 'target': 'http://127.0.0.1:42543', 'router': 'router__2Fuser_2Fgabbr_2F', 'service': 'service__2Fuser_2Fgabbr_2F'}}}}
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.254 JupyterHub proxy:340] Traefik route for /user/gabbr/: waiting to register
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.254 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.255 JupyterHub proxy:326] Traefik route for /user/gabbr/: service__2Fuser_2Fgabbr_2F@file not yet in service
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.445 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.447 JupyterHub proxy:372] 200 GET http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.447 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/routers/router__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.448 JupyterHub proxy:372] 200 GET http://localhost:8099/api/http/routers/router__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.448 JupyterHub proxy:357] Treafik route for /user/gabbr/: registered
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.448 JupyterHub users:768] Server gabbr is ready
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.449 JupyterHub log:191] 200 GET /hub/api/users/gabbr/server/progress?_xsrf=[secret] (gabbr@192.168.169.1) 960.74ms
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.506 JupyterHub scopes:877] Checking access to /hub/spawn-pending/gabbr via scope servers
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.506 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/gabbr via servers
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.506 JupyterHub log:191] 302 GET /hub/spawn-pending/gabbr -> /user/gabbr/ (gabbr@192.168.169.1) 2.02ms
Mar 12 12:20:40 tljh python3[89790]: [I 2024-03-12 12:20:40.549 ServerApp] 302 GET /user/gabbr/ -> /user/gabbr/lab? (@192.168.169.1) 0.47ms
Mar 12 12:20:40 tljh python3[89790]: [W 2024-03-12 12:20:40.606 ServerApp] 404 GET /user/gabbr/lab (@192.168.169.1) 13.01ms