AD with NFS automount home: Login works but landing page yields 404

JupyterHub The Littlest JupyterHub
, ,
1

Hello!

I’m having issues with my TLJH setup. The requirements are a little bit special (authentication through AD with auto-mounted home directories through NFS).
I can log in on the machine (ssh) and the home-directory is mounted correctly. I can log in to JupyterHub but after starting the server i am redirected to /user/<username>/lab which yields a 404.
I can access /hub/admin. I can start and stop the server but seem to be unable to access it. Any ideas, hints, pointers, links to relevant sections in the documentation?

The installation worked before i switched to LDAPAuthenticator - so i guess all the single parts work.

I tried setting c.Spawner.notebook_dir = '~' but that didn’t help.

I would like to either make the home directory or ~/jupyter the main Notebook directory.

This is my configuration:

import os
from glob import glob

from tljh import configurer
from tljh.config import CONFIG_DIR, INSTALL_PREFIX, USER_ENV_PREFIX
from jupyterhub.spawner import LocalProcessSpawner
from tljh.utils import get_plugin_manager

c = get_config()  # noqa
# c.JupyterHub.spawner_class = LocalProcessSpawner # this is the default

# leave users running when the Hub restarts
c.JupyterHub.cleanup_servers = False

# Use a high port so users can try this on machines with a JupyterHub already present
c.JupyterHub.hub_port = 15001

c.TraefikProxy.should_start = False

dynamic_conf_file_path = os.path.join(INSTALL_PREFIX, "state", "rules", "rules.toml")
c.TraefikFileProviderProxy.dynamic_config_file = dynamic_conf_file_path
c.JupyterHub.proxy_class = "traefik_file"

c.SystemdSpawner.extra_paths = [os.path.join(USER_ENV_PREFIX, "bin")]
c.SystemdSpawner.default_shell = "/bin/bash"
# Drop the '-singleuser' suffix present in the default template
c.SystemdSpawner.unit_name_template = "jupyter-{USERNAME}"

tljh_config = configurer.load_config()
configurer.apply_config(tljh_config, c)

# Let TLJH hooks modify `c` if they want

# Call our custom configuration plugin
pm = get_plugin_manager()
pm.hook.tljh_custom_jupyterhub_config(c=c)

# Load arbitrary .py config files if they exist.
# This is our escape hatch
extra_configs = sorted(glob(os.path.join(CONFIG_DIR, "jupyterhub_config.d", "*.py")))
for ec in extra_configs:
    load_subconfig(ec)

# get authentication information through LDAP
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = 'some.domain.local'
c.LDAPAuthenticator.lookup_dn = True
c.LDAPAuthenticator.lookup_dn_search_filter = '({login_attr}={login})'
c.LDAPAuthenticator.lookup_dn_search_user = 'aduser@domain.local' # TODO
c.LDAPAuthenticator.lookup_dn_search_password = '<password>'    # TODO
c.LDAPAuthenticator.user_search_base = 'ou=users,ou=domain,dc=domain,dc=local'
c.LDAPAuthenticator.user_attribute = 'sAMAccountName'
c.LDAPAuthenticator.lookup_dn_user_dn_attribute = 'cn'
c.LDAPAuthenticator.escape_userdn = False
c.LDAPAuthenticator.bind_dn_template = '{username}'
c.LDAPAuthenticator.use_lookup_dn_username = False
#c.LDAPAuthenticator.create

# create JupyterHub files within homedir
c.JupyterHub.log_level = 'DEBUG' # 'DEBUG' is the default for some reason?
# c.Spawner.notebook_dir = '~'
# c.Spawner.default_url = '/tree/home/{username}'
c.Spawner.debug = True # no change in logging output
# c.JupyterHub.allow_named_servers
c.Authenticator.admin_users = ('gabbr',)
2

Could you share the logs from JupyterHub and if available from single user server? Without logs it is really hard to find the issue. Cheers!

3

of course: here’s the debug logs of a restart of the service with the reproduced error:

Mar 12 12:20:34 tljh systemd[1]: Stopping jupyterhub.service...
Mar 12 12:20:34 tljh python3[89773]: [C 2024-03-12 12:20:34.356 JupyterHub app:3336] Received signal SIGTERM, initiating shutdown...
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.356 JupyterHub app:2976] Cleaning up 1 services...
Mar 12 12:20:34 tljh python3[89773]: [D 2024-03-12 12:20:34.356 JupyterHub service:436] Stopping service cull-idle
Mar 12 12:20:34 tljh python3[89773]: [D 2024-03-12 12:20:34.356 JupyterHub spawner:1777] Interrupting 89777
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:2988] Leaving single-user servers running
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:2996] I didn't start the proxy, I can't clean it up
Mar 12 12:20:34 tljh python3[89773]: [I 2024-03-12 12:20:34.456 JupyterHub app:3013] ...done
Mar 12 12:20:34 tljh systemd[1]: jupyterhub.service: Deactivated successfully.
Mar 12 12:20:34 tljh systemd[1]: Stopped jupyterhub.service.
Mar 12 12:20:34 tljh systemd[1]: jupyterhub.service: Consumed 1.171s CPU time.
Mar 12 12:20:34 tljh systemd[1]: Started jupyterhub.service.
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2859] Running JupyterHub version 4.0.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.0.2
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.393 JupyterHub app:2889] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.401 JupyterHub app:1664] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.401 JupyterHub app:1833] Connecting to db: sqlite:///jupyterhub.sqlite
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.412 JupyterHub orm:1016] database schema version found: 0eee8c825d24
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.417 JupyterHub orm:1016] database schema version found: 0eee8c825d24
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.454 JupyterHub app:2104] Loading roles into database
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.479 JupyterHub app:2343] Purging expired APITokens
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.480 JupyterHub app:2343] Purging expired OAuthCodes
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.482 JupyterHub app:2179] Loading role assignments from config
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.482 JupyterHub app:2250] Assigning 2 allowed_users to the user role
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.494 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.494 JupyterHub proxy:510] Setting up traefik's dynamic config...
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.506 JupyterHub fileprovider:95] Writing dynamic config {'http': {'routers': {'route_api': {'rule': 'Host(`localhost`) && PathPrefix(`/api`)', 'entryPoints': ['auth_api'], 'service': 'api@internal', 'middlewares': ['auth_api']}, 'router__2F': {'service': 'service__2F', 'rule': 'PathPrefix(`/`)', 'entryPoints': ['http']}}, 'middlewares': {'auth_api': {'basicAuth': {'users': ['api_admin:$apr1$GoQjK9yA$vJiIBIoiXJfM9QUxUbxuZ.']}}}, 'services': {'service__2F': {'loadBalancer': {'passHostHeader': True, 'servers': [{'url': 'http://127.0.0.1:15001'}]}}}}, 'jupyterhub': {'routes': {'router__2F': {'routespec': '/', 'target': 'http://127.0.0.1:15001', 'router': 'router__2F', 'service': 'service__2F', 'data': {'hub': True}}}}}
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.506 JupyterHub app:2502] Initializing spawners
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.508 JupyterHub app:2633] Loaded users:
Mar 12 12:20:35 tljh python3[89783]:     
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.508 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/overview
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.509 JupyterHub app:2928] Initialized 0 spawners in 0.015 seconds
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.514 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.twenty_four_hours
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.514 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.seven_days
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.515 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.thirty_days
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.515 JupyterHub app:3142] Not starting proxy
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.516 JupyterHub proxy:372] 200 GET http://localhost:8099/api/overview
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.516 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/entrypoints/http
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.517 JupyterHub proxy:372] 200 GET http://localhost:8099/api/entrypoints/http
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub app:3178] Hub API listening on http://127.0.0.1:15001/hub/
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub app:3189] Starting managed service cull-idle
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.517 JupyterHub service:385] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.518 JupyterHub service:133] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub spawner:1384] Polling subprocess every 30s
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub proxy:389] Fetching routes to check
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub proxy:392] Checking routes
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.519 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://127.0.0.1:15001/hub/
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.519 JupyterHub app:2852] It took 0.188 seconds for the Hub to start
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.610 JupyterHub base:299] Recording first activity for <APIToken('c378...', service='cull-idle', client_id='jupyterhub')>
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.615 JupyterHub log:191] 200 GET /hub/api/ (cull-idle@127.0.0.1) 6.44ms
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.617 JupyterHub scopes:877] Checking access to /hub/api/users via scope list:users
Mar 12 12:20:35 tljh python3[89783]: [D 2024-03-12 12:20:35.617 JupyterHub scopes:677] Unrestricted access to /hub/api/users via list:users
Mar 12 12:20:35 tljh python3[89783]: [I 2024-03-12 12:20:35.622 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 6.07ms
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.021 JupyterHub ldapauthenticator:256] Looking up user with:
Mar 12 12:20:38 tljh python3[89783]:         search_base = 'ou=users,ou=domain,dc=domain,dc=local'
Mar 12 12:20:38 tljh python3[89783]:         search_filter = '(sAMAccountName=gabbr)'
Mar 12 12:20:38 tljh python3[89783]:         attributes = 'sAMAccountName'
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.023 JupyterHub ldapauthenticator:379] Attempting to bind Firstname Lastname with Firstname Lastname
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.148 JupyterHub ldapauthenticator:392] Status of user bind Firstname Lastname with Firstname Lastname : True
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.151 JupyterHub roles:281] Assigning default role to User gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:587] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:591] Setting cookie for gabbr: jupyterhub-hub-login
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub base:587] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.154 JupyterHub base:837] User logged in: gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.154 JupyterHub user:431] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for gabbr:
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.155 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn (gabbr@192.168.169.1) 289.62ms
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.208 JupyterHub scopes:877] Checking access to /hub/spawn via scope servers
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.208 JupyterHub scopes:677] Unrestricted access to /hub/spawn via servers
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub pages:217] Triggering spawn with default options for gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:961] Initiating spawn for gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:965] 0/100 concurrent spawns
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.209 JupyterHub base:970] 0 active servers
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.228 JupyterHub provider:659] Creating oauth client jupyterhub-user-gabbr
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.239 JupyterHub user:794] Calling Spawner.start for gabbr
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.241 JupyterHub spawner:1689] Spawning jupyterhub-singleuser
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.253 JupyterHub spawner:1384] Polling subprocess every 30s
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.899 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.900 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.903 ServerApp] jupyterhub | extension was successfully linked.
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.923 JupyterHub log:191] 200 GET /hub/api (@127.0.0.1) 0.77ms
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.923 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.924 ServerApp] jupyterhub | extension was successfully loaded.
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Serving notebooks from local directory: /home/gabbr/jupyter
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Jupyter Server 2.12.5 is running at:
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] http://127.0.0.1:42543/user/gabbr/lab?token=...
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp]     http://127.0.0.1:42543/user/gabbr/lab?token=...
Mar 12 12:20:38 tljh python3[89790]: [I 2024-03-12 12:20:38.927 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.935 JupyterHub base:299] Recording first activity for <APIToken('c200...', user='gabbr', client_id='jupyterhub')>
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.940 JupyterHub scopes:877] Checking access to /hub/api/users/gabbr/activity via scope users:activity
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.940 JupyterHub scopes:690] Argument-based access to /hub/api/users/gabbr/activity via users:activity
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.941 JupyterHub users:874] Activity for user gabbr: 2024-03-12T12:20:38.911270Z
Mar 12 12:20:38 tljh python3[89783]: [D 2024-03-12 12:20:38.941 JupyterHub users:892] Activity on server gabbr/: 2024-03-12T12:20:38.911270Z
Mar 12 12:20:38 tljh python3[89783]: [I 2024-03-12 12:20:38.945 JupyterHub log:191] 200 POST /hub/api/users/gabbr/activity (gabbr@127.0.0.1) 11.94ms
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.211 JupyterHub log:191] 302 GET /hub/spawn -> /hub/spawn-pending/gabbr (gabbr@192.168.169.1) 1013.18ms
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.266 JupyterHub scopes:877] Checking access to /hub/spawn-pending/gabbr via scope servers
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.266 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/gabbr via servers
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.266 JupyterHub pages:398] gabbr is pending spawn
Mar 12 12:20:39 tljh python3[89783]: [I 2024-03-12 12:20:39.287 JupyterHub log:191] 200 GET /hub/spawn-pending/gabbr (gabbr@192.168.169.1) 23.20ms
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.489 JupyterHub scopes:877] Checking access to /hub/api/users/gabbr/server/progress via scope read:servers
Mar 12 12:20:39 tljh python3[89783]: [D 2024-03-12 12:20:39.489 JupyterHub scopes:677] Unrestricted access to /hub/api/users/gabbr/server/progress via read:servers
Mar 12 12:20:40 tljh python3[89790]: [I 2024-03-12 12:20:40.252 ServerApp] 302 GET /user/gabbr/ -> /user/gabbr/lab? (@127.0.0.1) 0.43ms
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.252 JupyterHub utils:278] Server at http://127.0.0.1:42543/user/gabbr/ responded with 302
Mar 12 12:20:40 tljh python3[89783]: [W 2024-03-12 12:20:40.252 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.252 JupyterHub base:990] User gabbr took 2.044 seconds to start
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.253 JupyterHub proxy:330] Adding user gabbr to proxy /user/gabbr/ => http://127.0.0.1:42543
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.253 JupyterHub fileprovider:95] Writing dynamic config {'http': {'routers': {'route_api': {'rule': 'Host(`localhost`) && PathPrefix(`/api`)', 'entryPoints': ['auth_api'], 'service': 'api@internal', 'middlewares': ['auth_api']}, 'router__2F': {'service': 'service__2F', 'rule': 'PathPrefix(`/`)', 'entryPoints': ['http']}, 'router__2Fuser_2Fgabbr_2F': {'service': 'service__2Fuser_2Fgabbr_2F', 'rule': '( PathPrefix(`/user/gabbr/`) || Path(`/user/gabbr`) )', 'entryPoints': ['http']}}, 'middlewares': {'auth_api': {'basicAuth': {'users': ['api_admin:$apr1$GoQjK9yA$vJiIBIoiXJfM9QUxUbxuZ.']}}}, 'services': {'service__2F': {'loadBalancer': {'passHostHeader': True, 'servers': [{'url': 'http://127.0.0.1:15001'}]}}, 'service__2Fuser_2Fgabbr_2F': {'loadBalancer': {'servers': [{'url': 'http://127.0.0.1:42543'}], 'passHostHeader': True}}}}, 'jupyterhub': {'routes': {'router__2F': {'routespec': '/', 'target': 'http://127.0.0.1:15001', 'router': 'router__2F', 'service': 'service__2F', 'data': {'hub': True}}, 'router__2Fuser_2Fgabbr_2F': {'data': {'user': 'gabbr', 'server_name': ''}, 'routespec': '/user/gabbr/', 'target': 'http://127.0.0.1:42543', 'router': 'router__2Fuser_2Fgabbr_2F', 'service': 'service__2Fuser_2Fgabbr_2F'}}}}
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.254 JupyterHub proxy:340] Traefik route for /user/gabbr/: waiting to register
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.254 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.255 JupyterHub proxy:326] Traefik route for /user/gabbr/: service__2Fuser_2Fgabbr_2F@file not yet in service
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.445 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.447 JupyterHub proxy:372] 200 GET http://localhost:8099/api/http/services/service__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.447 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/http/routers/router__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.448 JupyterHub proxy:372] 200 GET http://localhost:8099/api/http/routers/router__2Fuser_2Fgabbr_2F@file
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.448 JupyterHub proxy:357] Treafik route for /user/gabbr/: registered
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.448 JupyterHub users:768] Server gabbr is ready
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.449 JupyterHub log:191] 200 GET /hub/api/users/gabbr/server/progress?_xsrf=[secret] (gabbr@192.168.169.1) 960.74ms
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.506 JupyterHub scopes:877] Checking access to /hub/spawn-pending/gabbr via scope servers
Mar 12 12:20:40 tljh python3[89783]: [D 2024-03-12 12:20:40.506 JupyterHub scopes:677] Unrestricted access to /hub/spawn-pending/gabbr via servers
Mar 12 12:20:40 tljh python3[89783]: [I 2024-03-12 12:20:40.506 JupyterHub log:191] 302 GET /hub/spawn-pending/gabbr -> /user/gabbr/ (gabbr@192.168.169.1) 2.02ms
Mar 12 12:20:40 tljh python3[89790]: [I 2024-03-12 12:20:40.549 ServerApp] 302 GET /user/gabbr/ -> /user/gabbr/lab? (@192.168.169.1) 0.47ms
Mar 12 12:20:40 tljh python3[89790]: [W 2024-03-12 12:20:40.606 ServerApp] 404 GET /user/gabbr/lab (@192.168.169.1) 13.01ms
4

There is nothing obviously wrong in your logs. Can you make sure that you installed jupyterlab and notebook in the TLJH environment?

5

thank you so much for your help!

so, this looks suspicious. i did install TLJH through the official curl | python3 instructions, though.

do i have to manually install jupyterlab and notebook? is this a bug (in the installer)?

root@tljh:~# /opt/tljh/hub/bin/pip3 list
Package                          Version
-------------------------------- ---------
aiohttp                          3.9.3
aiosignal                        1.3.1
alembic                          1.13.1
async-generator                  1.10
async-timeout                    4.0.3
attrs                            23.2.0
backoff                          2.2.1
bcrypt                           4.1.2
certifi                          2024.2.2
certipy                          0.1.3
cffi                             1.16.0
charset-normalizer               3.3.2
cryptography                     42.0.5
escapism                         1.0.1
frozenlist                       1.4.1
greenlet                         3.0.3
idna                             3.6
Jinja2                           3.1.3
jsonschema                       4.21.1
jsonschema-specifications        2023.12.1
jupyter-telemetry                0.1.0
jupyterhub                       4.0.2
jupyterhub-firstuseauthenticator 1.0.0
jupyterhub-idle-culler           1.3.0
jupyterhub-ldapauthenticator     1.3.2
jupyterhub-nativeauthenticator   1.2.0
jupyterhub-systemdspawner        1.0.1
jupyterhub-tmpauthenticator      1.0.0
jupyterhub-traefik-proxy         1.1.0
ldap3                            2.9.1
Mako                             1.3.2
MarkupSafe                       2.1.5
multidict                        6.0.5
oauthenticator                   16.2.1
oauthlib                         3.2.2
onetimepass                      1.0.1
packaging                        23.2
pamela                           1.1.0
passlib                          1.7.4
pip                              24.0
pluggy                           1.4.0
prometheus_client                0.20.0
pyasn1                           0.5.1
pycparser                        2.21
pycurl                           7.45.3
pyOpenSSL                        24.0.0
python-dateutil                  2.8.2
python-json-logger               2.0.7
referencing                      0.33.0
requests                         2.31.0
rpds-py                          0.18.0
ruamel.yaml                      0.17.40
ruamel.yaml.clib                 0.2.8
setuptools                       59.6.0
six                              1.16.0
SQLAlchemy                       2.0.27
the-littlest-jupyterhub          1.0.0
toml                             0.10.2
tornado                          6.4
traitlets                        5.14.1
typing_extensions                4.10.0
urllib3                          2.2.1
yarl                             1.9.4
6

I dont know much about TLJH packaging. But looking here, apparently we will find single user environment at /opt/tljh/user. It is normal that JupyterHub do not contain JupyterLab and/or notebook

7

ah, ok. so it’s all there:

root@tljh:~# /opt/tljh/user/bin/pip3 list
Package                   Version
------------------------- ---------------
alembic                   1.13.1
anyio                     4.3.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-generator           1.10
async-lru                 2.0.4
attrs                     23.2.0
Babel                     2.14.0
beautifulsoup4            4.12.3
bleach                    6.1.0
brotlipy                  0.7.0
certifi                   2022.12.7
certipy                   0.1.3
cffi                      1.15.1
charset-normalizer        3.1.0
colorama                  0.4.6
comm                      0.2.1
conda                     23.1.0
conda-package-handling    2.0.2
conda_package_streaming   0.7.0
cryptography              40.0.1
debugpy                   1.8.1
decorator                 5.1.1
defusedxml                0.7.1
exceptiongroup            1.2.0
executing                 2.0.1
fastjsonschema            2.19.1
fqdn                      1.5.1
greenlet                  3.0.3
h11                       0.14.0
httpcore                  1.0.4
httpx                     0.27.0
idna                      3.4
ipykernel                 6.29.3
ipython                   8.22.1
ipywidgets                8.1.2
isoduration               20.11.0
jedi                      0.19.1
Jinja2                    3.1.3
json5                     0.9.17
jsonpointer               2.4
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
jupyter_client            8.6.0
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.3
jupyter-resource-usage    1.0.1
jupyter_server            2.12.5
jupyter_server_terminals  0.5.2
jupyter-telemetry         0.1.0
jupyterhub                4.0.2
jupyterlab                4.1.2
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.3
jupyterlab_widgets        3.0.10
libmambapy                1.4.1
Mako                      1.3.2
mamba                     1.4.1
MarkupSafe                2.1.5
matplotlib-inline         0.1.6
mistune                   3.0.2
nbclient                  0.9.0
nbconvert                 7.16.1
nbformat                  5.9.2
nbgitpuller               1.2.0
nest-asyncio              1.6.0
notebook                  7.1.1
notebook_shim             0.2.4
oauthlib                  3.2.2
overrides                 7.7.0
packaging                 23.2
pamela                    1.1.0
pandocfilters             1.5.1
parso                     0.8.3
pexpect                   4.9.0
pip                       24.0
platformdirs              4.2.0
pluggy                    1.0.0
prometheus_client         0.20.0
prompt-toolkit            3.0.43
psutil                    5.9.8
ptyprocess                0.7.0
pure-eval                 0.2.2
pycosat                   0.6.4
pycparser                 2.21
Pygments                  2.17.2
pyOpenSSL                 23.1.1
PySocks                   1.7.1
python-dateutil           2.8.2
python-json-logger        2.0.7
PyYAML                    6.0.1
pyzmq                     25.1.2
referencing               0.33.0
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.18.0
ruamel.yaml               0.17.21
ruamel.yaml.clib          0.2.7
Send2Trash                1.8.2
setuptools                65.6.3
six                       1.16.0
sniffio                   1.3.1
soupsieve                 2.5
SQLAlchemy                2.0.27
stack-data                0.6.3
terminado                 0.18.0
tinycss2                  1.2.1
tomli                     2.0.1
toolz                     0.12.0
tornado                   6.4
tqdm                      4.65.0
traitlets                 5.14.1
types-python-dateutil     2.8.19.20240106
typing_extensions         4.10.0
uri-template              1.3.0
urllib3                   1.26.15
wcwidth                   0.2.13
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
wheel                     0.40.0
widgetsnbextension        4.0.10
zstandard                 0.19.0

i added this line:

Environment=PYTHONPATH=/opt/tljh/user/lib/python3.10/site-packages

to the systemd service since it reported that it was unable to from jupyter_core import paths – which i found pretty odd

8

I am still wondering what goes wrong here. There are no error messages in the logs and I have no idea where to look (and for what exactly).

/user/${USERNAME} redirects (302) to /user/${USERNAME}/lab which 404s. Is this a JupyterHub bug? Or a Spawner bug? Or my misconfiguration?

The directories are being mounted correctly, the Spawner is started as the user that logs in and the lack of error messages in the logs except for the suspicious:
Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2 leave me wondering.

Any help or pointers or hints whatsoever are deeply appreciated. For more context, here is logging output with c.Spawner.args = ['--debug',] set:

Mar 20 13:14:46 jupyter systemd[1]: Started jupyterhub.service.
Mar 20 13:14:47 jupyter python3[1300]: [W 2024-03-20 13:14:47.093 JupyterHub app:2581] gabbr appears to have stopped while the Hub was down
Mar 20 13:14:47 jupyter python3[1300]: [W 2024-03-20 13:14:47.110 JupyterHub proxy:468] Deleting stale route /user/gabbr/
Mar 20 13:14:50 jupyter python3[1300]: [W 2024-03-20 13:14:50.928 JupyterHub log:191] 424 GET /hub/user/gabbr/lab (gabbr@10.11.12.13) 27.65ms
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.183 ServerApp] Searching ['/home/gabbr/.jupyter', '/opt/tljh/hub/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.183 ServerApp] Looking for jupyter_config in /etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.183 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.183 ServerApp] Looking for jupyter_config in /opt/tljh/hub/etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.183 ServerApp] Looking for jupyter_config in /home/gabbr/.jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.184 ServerApp] Looking for jupyter_server_config in /etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.184 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.184 ServerApp] Looking for jupyter_server_config in /opt/tljh/hub/etc/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.184 ServerApp] Looking for jupyter_server_config in /home/gabbr/.jupyter
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.187 ServerApp] Paths used for configuration of jupyter_server_config:
Mar 20 13:14:53 jupyter python3[1308]:             /etc/jupyter/jupyter_server_config.json
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.187 ServerApp] Paths used for configuration of jupyter_server_config:
Mar 20 13:14:53 jupyter python3[1308]:             /usr/local/etc/jupyter/jupyter_server_config.json
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.187 ServerApp] Paths used for configuration of jupyter_server_config:
Mar 20 13:14:53 jupyter python3[1308]:             /opt/tljh/hub/etc/jupyter/jupyter_server_config.json
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.187 ServerApp] Paths used for configuration of jupyter_server_config:
Mar 20 13:14:53 jupyter python3[1308]:             /home/gabbr/.jupyter/jupyter_server_config.json
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.191 ServerApp] Extension package jupyterhub took 0.0000s to import
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.192 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.192 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.196 JupyterHubSingleUser] Config changed: {'ServerApp': {'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'root_dir': '/home/gabbr/jupyter', 'port': 54931, 'ip': '127.0.0.1', 'base_url': '/user/gabbr/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': '', 'log_level': 'DEBUG'}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/opt/tljh/user/lib/python3.10/site-packages/jupyterhub/singleuser/templates']}>}, 'ExtensionApp': {'log_level': 'DEBUG'}}
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.196 ServerApp] jupyterhub | extension was successfully linked.
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.197 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'root_dir': '/home/gabbr/jupyter', 'port': 54931, 'ip': '127.0.0.1', 'base_url': '/user/gabbr/', 'default_url': '/lab', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/opt/tljh/user/lib/python3.10/site-packages/jupyterhub/singleuser/templates']}>}}
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.197 ServerApp] Raising open file limit: soft 1024->4096; hard 524288->524288
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.216 JupyterHubSingleUser] jupyterhub and jupyterhub-singleuser both on version 4.0.2
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.216 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.217 ServerApp] jupyterhub | extension was successfully loaded.
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.220 ServerApp] Serving notebooks from local directory: /home/gabbr/jupyter
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.220 ServerApp] Jupyter Server 2.12.5 is running at:
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.220 ServerApp] http://127.0.0.1:54931/user/gabbr/lab?token=...
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.220 ServerApp]     http://127.0.0.1:54931/user/gabbr/lab?token=...
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.220 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Mar 20 13:14:53 jupyter python3[1308]: [D 2024-03-20 13:14:53.225 JupyterHubSingleUser] Notifying Hub of activity 2024-03-20T13:14:53.203468Z
Mar 20 13:14:53 jupyter python3[1308]: [I 2024-03-20 13:14:53.633 ServerApp] 302 GET /user/gabbr/ -> /user/gabbr/lab? (@127.0.0.1) 0.46ms
Mar 20 13:14:53 jupyter python3[1300]: [W 2024-03-20 13:14:53.633 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 4.0.2
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.220 ServerApp] No user identified
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.220 ServerApp] Using contents: services/contents
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.231 ServerApp] Path favicon.ico served from /opt/tljh/user/lib/python3.10/site-packages/jupyter_server/static/favicon.ico
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.232 ServerApp] Path style/bootstrap.min.css served from /opt/tljh/user/lib/python3.10/site-packages/jupyter_server/static/style/bootstrap.min.css
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.232 ServerApp] Path style/bootstrap-theme.min.css served from /opt/tljh/user/lib/python3.10/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.232 ServerApp] Path style/index.css served from /opt/tljh/user/lib/python3.10/site-packages/jupyter_server/static/style/index.css
Mar 20 13:14:54 jupyter python3[1308]: [D 2024-03-20 13:14:54.232 ServerApp] Path logo/logo.png served from /opt/tljh/user/lib/python3.10/site-packages/jupyter_server/static/logo/logo.png
Mar 20 13:14:54 jupyter python3[1308]: [W 2024-03-20 13:14:54.232 ServerApp] 404 GET /user/gabbr/lab?redirects=1 (@10.11.12.13) 12.73ms
9

This suggests the problem is on the singleuser side since JupyterHub has successfully launched the singleuser server. Does it work if you use local home directories instead of auto-mounted NFS? Can you also try removing all configuration apart from LDAPAuthenticator and seeing if that changes things?

I suspect this indicates a deeper problem. This shouldn’t be necessary, is it possible you’ve got a corrupted Python environment, or some environment variables set somewhere that change how Python behaves?

10

Can you try setting the env var JUPYTERHUB_SINGLEUSER_EXTENSION=0 in the single user environment?

11

Thanks for your quick reply!

I dug deeper and found out the following things:

  • simply changing the desired spawner to LocalProcessSpawner produces ModuleNotFoundError: No module named 'jupyter_core'. That’s why I added the PYTHONPATH env-var to the systemd system process.
  • Changing auth.type to LDAPAuthenticator from an otherwise working setup yields a 500 Internal Server Error since the Spawner (IIUC) tries to a) create a new user jupyter-{USERNAME} together with a home-directory. Setting c.SystemdSpawner.username_template or c.SystemdSpawner.unit_name_template seem to have no effect.

This is the content of /opt/tljh/config/jupyterhub_config.d/10-spawner.py with seemingly no effect (my AD/LDAP config in /opt/tljh/config/jupyterhub_config.d/05-auth.py seems to work, though):

c.SystemdSpawner.unit_name_template = 'tljh-{USERNAME}'
c.SystemdSpawner.username_template = '{USERNAME}'
c.Spawner.notebook_dir = '~/jupyter'

Mar 21 15:04:03 jupyter-test python3[4110]: [W 2024-03-21 15:04:03.829 JupyterHub _version:38] Single-user server has no version header, which means it is likely < 0.8. Expected 4.1.0
Mar 21 15:04:09 jupyter-test useradd[4121]: new group: name=jupyter-gabbr, GID=1004
Mar 21 15:04:09 jupyter-test useradd[4121]: new user: name=jupyter-gabbr, UID=1002, GID=1004, home=/home/jupyter-gabbr, shell=/bin/sh, from=none
Mar 21 15:04:09 jupyter-test python3[4121]: useradd: cannot create directory /home/jupyter-gabbr
Mar 21 15:04:09 jupyter-test useradd[4121]: failed adding user 'jupyter-gabbr', data deleted
Mar 21 15:04:09 jupyter-test python3[4110]: [E 2024-03-21 15:04:09.651 JupyterHub user:887] Unhandled error starting gabbr's server: Command '['useradd', '--create-home', 'jupyter-gabbr']' returned non-zero exit status 12.
Mar 21 15:04:09 jupyter-test python3[4110]:     Traceback (most recent call last):
Mar 21 15:04:09 jupyter-test python3[4110]:       File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 798, in spawn
Mar 21 15:04:09 jupyter-test python3[4110]:         f = maybe_future(spawner.start())
Mar 21 15:04:09 jupyter-test python3[4110]:       File "/opt/tljh/hub/lib/python3.10/site-packages/tljh/user_creating_spawner.py", line 26, in start
Mar 21 15:04:09 jupyter-test python3[4110]:         user.ensure_user(system_username)
Mar 21 15:04:09 jupyter-test python3[4110]:       File "/opt/tljh/hub/lib/python3.10/site-packages/tljh/user.py", line 28, in ensure_user
Mar 21 15:04:09 jupyter-test python3[4110]:         subprocess.check_call(["useradd", "--create-home", username])
Mar 21 15:04:09 jupyter-test python3[4110]:       File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
Mar 21 15:04:09 jupyter-test python3[4110]:         raise CalledProcessError(retcode, cmd)
Mar 21 15:04:09 jupyter-test python3[4110]:     subprocess.CalledProcessError: Command '['useradd', '--create-home', 'jupyter-gabbr']' returned non-zero exit status 12.
Mar 21 15:04:09 jupyter-test python3[4110]:     
Mar 21 15:04:09 jupyter-test python3[4110]: [E 2024-03-21 15:04:09.673 JupyterHub pages:314] Error starting server gabbr: Command '['useradd', '--create-home', 'jupyter-gabbr']' returned non-zero exit status 12.

Is it possible to get a process spawned as the user that just logged in? This is necessary since we need to re-use UIDs that own the auto-mounted NFS shares. It seemed as if LocalProcessSpawner were the right tool for that…

Is there some example configuration for my use-case (or something similar)?

Is it possible to display currently effective configuration (defaults, what ends up in config.yaml and valid settings form the jupyterhub_config.d/ directory?
Is there a (more or less complete) reference of possible configuration keys?

12

Can you try setting the env var JUPYTERHUB_SINGLEUSER_EXTENSION=0 in the single user environment?

How can I do that?

13

See this comment for setting custom env vars on spawner

14

TLJH is designed to be a self-contained easy-to-install distribution which automates a lot of the configuration. The downside is it’s sometimes harder (though still possible) to customise some components, such as switching away from the spawner. Needing to modify PYTHONPATH is probably one example of this, and suggests there are probably other issues. For example, site-packages is pretty important for a standard Python environment so if it was completely missing from your Python environment I’d expect almost nothing to work, but it looks like you’ve gotten at least as far as trying (and failing) to load jupyter_core. Another example of the self-contained nature is that TLJH uses a variant of the SystemdSpawner that overrides the username, which is why c.SystemdSpawner.username_template has no effect:

It might be worth trying to switch to the unmodified SystemdSpawner spawner instead of UserCreatingSpawner, and seeing if the available configuration options work.

If that still doesn’t work then we’re getting quite deep into TLJH, and we’ve lost most of the benefits of TLJH, so it might be worth creating a new server and manually installing JupyterHub (which will default to LocalProcessSpawner) using Quickstart — JupyterHub documentation , then configure LDAPAuthenticator.