Using a fresh install of JupyterHub 2.3.1. After successful login via the Hub, the single user server seems to spawn correctly, but then throws a 404 error when trying to access the file tree (or anything at all).
The SingleUserNotebookApp
has a log entry that says No user identified
and I think this may be part of the problem. I can’t figure out what it the root cause.
An interesting datapoint, if I downgrade the version of Jupyterhub to 1.4.0, it works as expected.
Here is the log output:
[D 2022-07-20 22:58:00.314 JupyterHub application:837] Looking for jupyterhub_config in /tmp/ic-runtime-assets/plain
[D 2022-07-20 22:58:00.314 JupyterHub application:858] Loaded config file: /tmp/ic-runtime-assets/plain/jupyterhub_config.py
[I 2022-07-20 22:58:00.315 JupyterHub app:2771] Running JupyterHub version 2.3.1
[I 2022-07-20 22:58:00.315 JupyterHub app:2801] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.3.1
[I 2022-07-20 22:58:00.315 JupyterHub app:2801] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.3.1
[I 2022-07-20 22:58:00.315 JupyterHub app:2801] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.3.1
[I 2022-07-20 22:58:00.319 JupyterHub app:1606] Loading cookie_secret from /tmp/ic-runtime-assets/plain/cookie_secret
[D 2022-07-20 22:58:00.319 JupyterHub app:1775] Connecting to db: sqlite:////tmp/ic-runtime-assets/plain/jupyterhub.sqlite
[D 2022-07-20 22:58:00.328 JupyterHub orm:953] database schema version found: 833da8570507
[I 2022-07-20 22:58:00.377 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-07-20 22:58:00.377 JupyterHub app:2024] Loading roles into database
[I 2022-07-20 22:58:00.383 JupyterHub app:1926] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-07-20 22:58:00.386 JupyterHub app:2283] Purging expired APITokens
[D 2022-07-20 22:58:00.387 JupyterHub app:2283] Purging expired OAuthCodes
[D 2022-07-20 22:58:00.388 JupyterHub app:2116] Loading role assignments from config
[D 2022-07-20 22:58:00.393 JupyterHub app:2429] Initializing spawners
[D 2022-07-20 22:58:00.394 JupyterHub app:2560] Loaded users:
[I 2022-07-20 22:58:00.394 JupyterHub app:2840] Initialized 0 spawners in 0.001 seconds
[W 2022-07-20 22:58:00.396 JupyterHub proxy:687] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2022-07-20 22:58:00.396 JupyterHub proxy:691] Starting proxy @ http://:8888
[D 2022-07-20 22:58:00.396 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8888', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2022-07-20 22:58:00.400 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
22:58:00.482 [ConfigProxy] info: Proxying http://*:8888 to (no default)
22:58:00.483 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2022-07-20 22:58:00.511 JupyterHub proxy:728] Proxy started and appears to be up
[D 2022-07-20 22:58:00.511 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2022-07-20 22:58:00.519 JupyterHub app:3089] Hub API listening on http://127.0.0.1:8081/hub/
[D 2022-07-20 22:58:00.519 JupyterHub proxy:343] Fetching routes to check
[D 2022-07-20 22:58:00.519 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
22:58:00.520 [ConfigProxy] info: 200 GET /api/routes
22:58:00.521 [ConfigProxy] info: 200 GET /api/routes
[D 2022-07-20 22:58:00.521 JupyterHub proxy:346] Checking routes
[I 2022-07-20 22:58:00.521 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
[D 2022-07-20 22:58:00.521 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
22:58:00.522 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
22:58:00.522 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
22:58:00.523 [ConfigProxy] info: 201 POST /api/routes/
[I 2022-07-20 22:58:00.523 JupyterHub app:3156] JupyterHub is now running at http://:8888
[D 2022-07-20 22:58:00.523 JupyterHub app:2764] It took 0.212 seconds for the Hub to start
[I 2022-07-20 22:58:12.544 JupyterHub log:189] 302 GET / -> /hub/ (@::ffff:172.18.0.1) 0.77ms
[I 2022-07-20 22:58:12.583 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:172.18.0.1) 0.38ms
[I 2022-07-20 22:58:12.653 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:172.18.0.1) 19.99ms
[D 2022-07-20 22:58:12.708 JupyterHub log:189] 200 GET /hub/static/css/style.min.css?v=bff49b4a161afb17ee3b71927ce7d6c4e5b0e4b9ef6f18ca3e356a05f29e69776d3a76aee167060dd2ae2ee62d3cfdcf203b4b0090b1423f7d629ea7daa3f9da (@::ffff:172.18.0.1) 0.82ms
[D 2022-07-20 22:58:12.721 JupyterHub log:189] 200 GET /hub/static/components/requirejs/require.js?v=bd1aa102bdb0b27fbf712b32cfcd29b016c272acf3d864ee8469376eaddd032cadcf827ff17c05a8c8e20061418fe58cf79947049f5c0dff3b4f73fcc8cad8ec (@::ffff:172.18.0.1) 0.44ms
[D 2022-07-20 22:58:12.722 JupyterHub log:189] 200 GET /hub/static/components/jquery/dist/jquery.min.js?v=f3de1813a4160f9239f4781938645e1589b876759cd50b7936dbd849a35c38ffaed53f6a61dbdd8a1cf43cf4a28aa9fffbfddeec9a3811a1bb4ee6df58652b31 (@::ffff:172.18.0.1) 0.60ms
[D 2022-07-20 22:58:12.723 JupyterHub log:189] 200 GET /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=a014e9acc78d10a0a7a9fbaa29deac6ef17398542d9574b77b40bf446155d210fa43384757e3837da41b025998ebfab4b9b6f094033f9c226392b800df068bce (@::ffff:172.18.0.1) 0.77ms
[D 2022-07-20 22:58:12.799 JupyterHub log:189] 200 GET /hub/logo (@::ffff:172.18.0.1) 0.54ms
[D 2022-07-20 22:58:12.851 JupyterHub log:189] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:172.18.0.1) 0.47ms
[D 2022-07-20 22:59:36.842 JupyterHub log:189] 200 GET /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 (@::ffff:172.18.0.1) 0.88ms
[D 2022-07-20 22:59:36.860 JupyterHub roles:454] Assigning default role to User max
[D 2022-07-20 22:59:36.866 JupyterHub base:565] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2022-07-20 22:59:36.866 JupyterHub base:569] Setting cookie for max: jupyterhub-hub-login
[D 2022-07-20 22:59:36.866 JupyterHub base:565] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-07-20 22:59:36.866 JupyterHub base:816] User logged in: max
[I 2022-07-20 22:59:36.867 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (max@::ffff:172.18.0.1) 24.60ms
[D 2022-07-20 22:59:36.921 JupyterHub user:399] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for max:
[I 2022-07-20 22:59:36.922 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (max@::ffff:172.18.0.1) 10.06ms
[D 2022-07-20 22:59:36.962 JupyterHub scopes:491] Checking access via scope servers
[D 2022-07-20 22:59:36.962 JupyterHub scopes:402] Argument-based access to /hub/spawn via servers
[D 2022-07-20 22:59:36.962 JupyterHub pages:215] Triggering spawn with default options for max
[D 2022-07-20 22:59:36.962 JupyterHub base:934] Initiating spawn for max
[D 2022-07-20 22:59:36.962 JupyterHub base:938] 0/100 concurrent spawns
[D 2022-07-20 22:59:36.962 JupyterHub base:943] 0 active servers
[D 2022-07-20 22:59:36.967 JupyterHub roles:477] Checking token permissions against requested role server
[I 2022-07-20 22:59:36.969 JupyterHub roles:482] Adding role server to token: <APIToken('b9d4...', user='max', client_id='jupyterhub')>
[I 2022-07-20 22:59:36.975 JupyterHub provider:607] Creating oauth client jupyterhub-user-max
[D 2022-07-20 22:59:36.986 JupyterHub user:728] Calling Spawner.start for max
[I 2022-07-20 22:59:36.987 JupyterHub spawner:1563] Spawning jupyterhub-singleuser
[D 2022-07-20 22:59:36.998 JupyterHub spawner:1258] Polling subprocess every 30s
[I 2022-07-20 22:59:37.527 SingleUserNotebookApp mixins:615] Starting jupyterhub single-user server version 2.3.1
[I 2022-07-20 22:59:37.527 SingleUserNotebookApp mixins:629] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.17.0
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:164] Searching ['/home/max', '/home/max/.jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:730] Looking for jupyter_config in /etc/jupyter
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:730] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:730] Looking for jupyter_config in /usr/etc/jupyter
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/max/.jupyter
[D 2022-07-20 22:59:37.534 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/max
[D 2022-07-20 22:59:37.535 SingleUserNotebookApp application:730] Looking for jupyter_server_config in /etc/jupyter
[D 2022-07-20 22:59:37.535 SingleUserNotebookApp application:730] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2022-07-20 22:59:37.535 SingleUserNotebookApp application:730] Looking for jupyter_server_config in /usr/etc/jupyter
[D 2022-07-20 22:59:37.535 SingleUserNotebookApp application:730] Looking for jupyter_server_config in /home/max/.jupyter
[D 2022-07-20 22:59:37.535 SingleUserNotebookApp application:730] Looking for jupyter_server_config in /home/max
[D 2022-07-20 22:59:37.536 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
/etc/jupyter/jupyter_server_config.json
[D 2022-07-20 22:59:37.536 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
/usr/local/etc/jupyter/jupyter_server_config.d/jupyterlab.json
/usr/local/etc/jupyter/jupyter_server_config.d/nbgitpuller.json
/usr/local/etc/jupyter/jupyter_server_config.d/notebook_shim.json
/usr/local/etc/jupyter/jupyter_server_config.json
[D 2022-07-20 22:59:37.537 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
/usr/etc/jupyter/jupyter_server_config.json
[D 2022-07-20 22:59:37.537 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
/home/max/.jupyter/jupyter_server_config.json
[D 2022-07-20 22:59:37.537 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
/home/max/jupyter_server_config.json
[W 2022-07-20 22:59:37.538 SingleUserNotebookApp manager:329]
[I 2022-07-20 22:59:37.539 SingleUserNotebookApp manager:340] nbgitpuller | extension was successfully linked.
[D 2022-07-20 22:59:37.626 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
/home/max/.jupyter/jupyter_notebook_config.json
[D 2022-07-20 22:59:37.627 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-20 22:59:37.627 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/usr/local/etc/jupyter/jupyter_notebook_config.d/nbgitpuller.json
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-20 22:59:37.627 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
/usr/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-20 22:59:37.627 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
/home/max/.jupyter/jupyter_notebook_config.json
[W 2022-07-20 22:59:37.630 SingleUserNotebookApp manager:329] The module 'nbdime' could not be found. Are you sure the extension is installed?
[W 2022-07-20 22:59:37.631 SingleUserNotebookApp manager:329]
[I 2022-07-20 22:59:37.631 SingleUserNotebookApp manager:340] notebook_shim | extension was successfully linked.
[I 2022-07-20 22:59:37.645 SingleUserNotebookApp manager:362] notebook_shim | extension was successfully loaded.
[I 2022-07-20 22:59:37.646 SingleUserNotebookApp manager:362] nbgitpuller | extension was successfully loaded.
[I 2022-07-20 22:59:37.646 SingleUserNotebookApp mixins:648] Starting jupyterhub-singleuser server version 2.3.1
[I 2022-07-20 22:59:37.648 JupyterHub log:189] 200 GET /hub/api (@127.0.0.1) 0.77ms
[D 2022-07-20 22:59:37.648 SingleUserNotebookApp _version:74] jupyterhub and jupyterhub-singleuser both on version 2.3.1
[I 2022-07-20 22:59:37.648 SingleUserNotebookApp serverapp:2671] Serving notebooks from local directory: /home
[I 2022-07-20 22:59:37.648 SingleUserNotebookApp serverapp:2671] Jupyter Server 1.17.0 is running at:
[I 2022-07-20 22:59:37.648 SingleUserNotebookApp serverapp:2671] http://127.0.0.1:49409/user/max/tree
[I 2022-07-20 22:59:37.648 SingleUserNotebookApp serverapp:2671] or http://127.0.0.1:49409/user/max/tree
[I 2022-07-20 22:59:37.648 SingleUserNotebookApp serverapp:2672] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2022-07-20 22:59:37.650 SingleUserNotebookApp mixins:597] Updating Hub with activity every 300 seconds
[D 2022-07-20 22:59:37.650 SingleUserNotebookApp mixins:559] Notifying Hub of activity 2022-07-20T22:59:37.639910Z
[D 2022-07-20 22:59:37.654 JupyterHub base:281] Recording first activity for <APIToken('b9d4...', user='max', client_id='jupyterhub')>
[D 2022-07-20 22:59:37.658 JupyterHub scopes:301] Authenticated with token <APIToken('b9d4...', user='max', client_id='jupyterhub')>
[D 2022-07-20 22:59:37.659 JupyterHub scopes:491] Checking access via scope users:activity
[D 2022-07-20 22:59:37.659 JupyterHub scopes:402] Argument-based access to /hub/api/users/max/activity via users:activity
[D 2022-07-20 22:59:37.660 JupyterHub users:859] Activity for user max: 2022-07-20T22:59:37.639910Z
[D 2022-07-20 22:59:37.660 JupyterHub users:877] Activity on server max/: 2022-07-20T22:59:37.639910Z
[I 2022-07-20 22:59:37.663 JupyterHub log:189] 200 POST /hub/api/users/max/activity (max@127.0.0.1) 11.73ms
[I 2022-07-20 22:59:37.964 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/max (max@::ffff:172.18.0.1) 1003.03ms
[D 2022-07-20 22:59:38.112 JupyterHub scopes:491] Checking access via scope servers
[D 2022-07-20 22:59:38.113 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/max via servers
[I 2022-07-20 22:59:38.113 JupyterHub pages:401] max is pending spawn
[I 2022-07-20 22:59:38.116 JupyterHub log:189] 200 GET /hub/spawn-pending/max (max@::ffff:172.18.0.1) 6.29ms
[D 2022-07-20 22:59:38.191 JupyterHub scopes:491] Checking access via scope read:servers
[D 2022-07-20 22:59:38.191 JupyterHub scopes:402] Argument-based access to /hub/api/users/max/server/progress via read:servers
[I 2022-07-20 22:59:38.417 SingleUserNotebookApp log:189] 302 GET /user/max/ -> /user/max/tree? (@127.0.0.1) 0.60ms
[D 2022-07-20 22:59:38.417 JupyterHub utils:230] Server at http://127.0.0.1:49409/user/max/ responded with 302
[D 2022-07-20 22:59:38.418 JupyterHub _version:74] jupyterhub and jupyterhub-singleuser both on version 2.3.1
[I 2022-07-20 22:59:38.418 JupyterHub base:963] User max took 1.455 seconds to start
[I 2022-07-20 22:59:38.418 JupyterHub proxy:286] Adding user max to proxy /user/max/ => http://127.0.0.1:49409
[D 2022-07-20 22:59:38.418 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/max
22:59:38.419 [ConfigProxy] info: Adding route /user/max -> http://127.0.0.1:49409
22:59:38.420 [ConfigProxy] info: Route added /user/max -> http://127.0.0.1:49409
22:59:38.420 [ConfigProxy] info: 201 POST /api/routes/user/max
[I 2022-07-20 22:59:38.420 JupyterHub users:753] Server max is ready
[I 2022-07-20 22:59:38.421 JupyterHub log:189] 200 GET /hub/api/users/max/server/progress (max@::ffff:172.18.0.1) 231.11ms
[D 2022-07-20 22:59:38.466 JupyterHub scopes:491] Checking access via scope servers
[D 2022-07-20 22:59:38.466 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/max via servers
[I 2022-07-20 22:59:38.466 JupyterHub log:189] 302 GET /hub/spawn-pending/max -> /user/max/ (max@::ffff:172.18.0.1) 2.11ms
[I 2022-07-20 22:59:38.508 SingleUserNotebookApp log:189] 302 GET /user/max/ -> /user/max/tree? (@::ffff:172.18.0.1) 0.56ms
[D 2022-07-20 22:59:38.551 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-20 22:59:38.551 SingleUserNotebookApp handlers:257] Using contents: services/contents
[D 2022-07-20 22:59:38.565 SingleUserNotebookApp handlers:857] Path favicon.ico served from /home/max/.local/lib/python3.8/site-packages/jupyter_server/static/favicon.ico
[D 2022-07-20 22:59:38.565 SingleUserNotebookApp handlers:857] Path style/bootstrap.min.css served from /home/max/.local/lib/python3.8/site-packages/jupyter_server/static/style/bootstrap.min.css
[D 2022-07-20 22:59:38.566 SingleUserNotebookApp handlers:857] Path style/bootstrap-theme.min.css served from /home/max/.local/lib/python3.8/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
[D 2022-07-20 22:59:38.566 SingleUserNotebookApp handlers:857] Path style/index.css served from /home/max/.local/lib/python3.8/site-packages/jupyter_server/static/style/index.css
[W 2022-07-20 22:59:38.566 SingleUserNotebookApp log:189] 404 GET /user/max/tree? (@::ffff:172.18.0.1) 15.62ms
[D 2022-07-20 22:59:38.607 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-20 22:59:38.608 SingleUserNotebookApp log:189] 200 GET /user/max/static/style/bootstrap.min.css?v=0e8a7fbd6de23ad6b27ab95802a0a0915af6693af612bc304d83af445529ce5d95842309ca3405d10f538d45c8a3a261b8cff78b4bd512dd9effb4109a71d0ab (@::ffff:172.18.0.1) 1.05ms
[D 2022-07-20 22:59:38.608 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-20 22:59:38.609 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-20 22:59:38.609 SingleUserNotebookApp log:189] 200 GET /user/max/static/style/bootstrap-theme.min.css?v=8b2f045cb5b4d5ad346f6e816aa2566829a4f5f2783ec31d80d46a57de8ac0c3d21fe6e53bcd8e1f38ac17fcd06d12088bc9b43e23b5d1da52d10c6b717b22b3 (@::ffff:172.18.0.1) 1.08ms
[D 2022-07-20 22:59:38.610 SingleUserNotebookApp log:189] 200 GET /user/max/static/style/index.css?v=30372e3246a801d662cf9e3f9dd656fa192eebde9054a2282449fe43919de9f0ee9b745d7eb49d3b0a5e56357912cc7d776390eddcab9dac85b77bdb17b4bdae (@::ffff:172.18.0.1) 1.16ms
[D 2022-07-20 22:59:38.718 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-20 22:59:38.719 SingleUserNotebookApp log:189] 200 GET /user/max/static/favicon.ico?v=50afa725b5de8b00030139d09b38620224d4e7dba47c07ef0e86d4643f30c9bfe6bb7e1a4a1c561aa32834480909a4b6fe7cd1e17f7159330b6b5914bf45a880 (@::ffff:172.18.0.1) 0.69ms
My config is quite simple…
import os
pjoin = os.path.join
runtime_dir = os.path.join("/tmp/ic-runtime-assets/plain")
c.JupyterHub.cookie_secret_file = pjoin(runtime_dir, "cookie_secret")
c.JupyterHub.db_url = pjoin(runtime_dir, "jupyterhub.sqlite")
c.Spawner.debug = True
c.Spawner.default_url = "/tree"
c.Spawner.notebook_dir = "/home"
Any guidance on how I might fix this would be greatly appreciated.