404 error from fresh install of jupyterhub

Hello,

I followed the instructions on: Quickstart — JupyterHub 2.3.1 documentation on a fresh computer / virtual environment and installed Jupyterhub. Trying to run it gives me a 404 error after logging in.

Screenshot 2022-07-06 180542

I have turned various debugging features on (via a config file in ~/jupyterhub) and pasted the output below. My initial guess is something to do with the “No user identified” but I’m not sure how I would go about to fix it.

Any help is much appreciated.

Sincerely,
Fen

(sparkfly) fen@fenix:~/jupyterhub$ sudo jupyterhub --debug
[D 2022-07-06 18:03:00.598 JupyterHub application:837] Looking for jupyterhub_config in /home/fen/jupyterhub
[D 2022-07-06 18:03:00.598 JupyterHub application:858] Loaded config file: /home/fen/jupyterhub/jupyterhub_config.py
[I 2022-07-06 18:03:00.599 JupyterHub app:2771] Running JupyterHub version 2.3.1
[I 2022-07-06 18:03:00.599 JupyterHub app:2801] Using Authenticator: jupyterhub.auth.PAMAuthenticator-2.3.1
[I 2022-07-06 18:03:00.599 JupyterHub app:2801] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-2.3.1
[I 2022-07-06 18:03:00.599 JupyterHub app:2801] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.3.1
[D 2022-07-06 18:03:00.600 JupyterHub app:2732] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2022-07-06 18:03:00.600 JupyterHub app:1606] Loading cookie_secret from /home/fen/jupyterhub/jupyterhub_cookie_secret
[D 2022-07-06 18:03:00.600 JupyterHub app:1775] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-07-06 18:03:00.608 JupyterHub orm:953] database schema version found: 833da8570507
[I 2022-07-06 18:03:00.632 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-07-06 18:03:00.633 JupyterHub app:2024] Loading roles into database
[I 2022-07-06 18:03:00.638 JupyterHub app:1926] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-07-06 18:03:00.640 JupyterHub app:2283] Purging expired APITokens
[D 2022-07-06 18:03:00.642 JupyterHub app:2283] Purging expired OAuthCodes
[D 2022-07-06 18:03:00.643 JupyterHub app:2116] Loading role assignments from config
[D 2022-07-06 18:03:00.648 JupyterHub app:2429] Initializing spawners
[D 2022-07-06 18:03:00.650 JupyterHub app:2560] Loaded users:

[I 2022-07-06 18:03:00.650 JupyterHub app:2840] Initialized 0 spawners in 0.001 seconds
[W 2022-07-06 18:03:00.651 JupyterHub proxy:687] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere e
lse...
[I 2022-07-06 18:03:00.651 JupyterHub proxy:691] Starting proxy @ http://:8000
[D 2022-07-06 18:03:00.651 JupyterHub proxy:692] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.
0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2022-07-06 18:03:00.651 JupyterHub proxy:610] Writing proxy pid file: jupyterhub-proxy.pid
18:03:00.773 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:03:00.775 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2022-07-06 18:03:00.952 JupyterHub proxy:728] Proxy started and appears to be up
[D 2022-07-06 18:03:00.957 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2022-07-06 18:03:00.986 JupyterHub app:3089] Hub API listening on http://127.0.0.1:8081/hub/
[D 2022-07-06 18:03:00.986 JupyterHub proxy:343] Fetching routes to check
[D 2022-07-06 18:03:00.986 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:03:00.987 [ConfigProxy] info: 200 GET /api/routes
18:03:00.990 [ConfigProxy] info: 200 GET /api/routes
[D 2022-07-06 18:03:00.991 JupyterHub proxy:346] Checking routes
[I 2022-07-06 18:03:00.991 JupyterHub proxy:431] Adding route for Hub: / => http://127.0.0.1:8081
[D 2022-07-06 18:03:00.991 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
18:03:00.994 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
18:03:00.995 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
18:03:00.996 [ConfigProxy] info: 201 POST /api/routes/
[I 2022-07-06 18:03:00.996 JupyterHub app:3156] JupyterHub is now running at http://:8000
[D 2022-07-06 18:03:00.997 JupyterHub app:2764] It took 0.402 seconds for the Hub to start
[I 2022-07-06 18:03:06.760 JupyterHub log:189] 302 GET /user/fen/tree -> /hub/user/fen/tree (@::ffff:10.9.1.1) 0.37ms
[D 2022-07-06 18:03:06.789 JupyterHub base:326] Refreshing auth for fen
[D 2022-07-06 18:03:06.791 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-07-06 18:03:06.791 JupyterHub scopes:402] Argument-based access to /hub/user/fen/tree via access:servers
[D 2022-07-06 18:03:06.791 JupyterHub user:399] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for fen:
[W 2022-07-06 18:03:06.807 JupyterHub log:189] 424 GET /hub/user/fen/tree (fen@::ffff:10.9.1.1) 23.13ms
[D 2022-07-06 18:03:06.906 JupyterHub log:189] 200 GET /hub/static/js/not_running.js?v=20220706180300 (@::ffff:10.9.1.1) 13.25ms
[D 2022-07-06 18:03:06.935 JupyterHub log:189] 200 GET /hub/static/js/utils.js?v=20220706180300 (@::ffff:10.9.1.1) 2.11ms
[D 2022-07-06 18:03:09.115 JupyterHub scopes:491] Checking access via scope servers
[D 2022-07-06 18:03:09.115 JupyterHub scopes:402] Argument-based access to /hub/spawn/fen via servers
[D 2022-07-06 18:03:09.115 JupyterHub pages:215] Triggering spawn with default options for fen
[D 2022-07-06 18:03:09.116 JupyterHub base:934] Initiating spawn for fen
[D 2022-07-06 18:03:09.116 JupyterHub base:938] 0/100 concurrent spawns
[D 2022-07-06 18:03:09.116 JupyterHub base:943] 0 active servers
[D 2022-07-06 18:03:09.119 JupyterHub roles:477] Checking token permissions against requested role server
[I 2022-07-06 18:03:09.120 JupyterHub roles:482] Adding role server to token: <APIToken('801d...', user='fen', client_id='jupyterhub')
[I 2022-07-06 18:03:09.135 JupyterHub provider:607] Creating oauth client jupyterhub-user-fen
[D 2022-07-06 18:03:09.191 JupyterHub user:728] Calling Spawner.start for fen
[I 2022-07-06 18:03:09.194 JupyterHub spawner:1563] Spawning jupyterhub-singleuser --debug
[D 2022-07-06 18:03:09.229 JupyterHub spawner:1258] Polling subprocess every 30s
[I 2022-07-06 18:03:09.633 SingleUserNotebookApp mixins:615] Starting jupyterhub single-user server version 2.3.1
[I 2022-07-06 18:03:09.633 SingleUserNotebookApp mixins:629] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.18.0
[D 2022-07-06 18:03:09.640 SingleUserNotebookApp application:183] Searching ['/home/fen', '/home/fen/.jupyter', '/home/fen/.local/etc/
upyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2022-07-06 18:03:09.640 SingleUserNotebookApp application:837] Looking for jupyter_config in /etc/jupyter
[D 2022-07-06 18:03:09.640 SingleUserNotebookApp application:837] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2022-07-06 18:03:09.640 SingleUserNotebookApp application:837] Looking for jupyter_config in /usr/etc/jupyter
[D 2022-07-06 18:03:09.640 SingleUserNotebookApp application:837] Looking for jupyter_config in /home/fen/.local/etc/jupyter
[D 2022-07-06 18:03:09.641 SingleUserNotebookApp application:837] Looking for jupyter_config in /home/fen/.jupyter
[D 2022-07-06 18:03:09.641 SingleUserNotebookApp application:837] Looking for jupyter_config in /home/fen
[D 2022-07-06 18:03:09.641 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /etc/jupyter
[D 2022-07-06 18:03:09.641 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2022-07-06 18:03:09.641 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /usr/etc/jupyter
[D 2022-07-06 18:03:09.642 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /home/fen/.local/etc/jupyter
[D 2022-07-06 18:03:09.642 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /home/fen/.jupyter
[D 2022-07-06 18:03:09.642 SingleUserNotebookApp application:837] Looking for jupyter_server_config in /home/fen
[D 2022-07-06 18:03:09.643 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /etc/jupyter/jupyter_server_config.json
[D 2022-07-06 18:03:09.644 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /usr/local/etc/jupyter/jupyter_server_config.d/jupyterlab.json
        /usr/local/etc/jupyter/jupyter_server_config.d/notebook_shim.json
        /usr/local/etc/jupyter/jupyter_server_config.json
[D 2022-07-06 18:03:09.644 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /usr/etc/jupyter/jupyter_server_config.json
[D 2022-07-06 18:03:09.644 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /home/fen/.local/etc/jupyter/jupyter_server_config.d/notebook_shim.json
        /home/fen/.local/etc/jupyter/jupyter_server_config.json
[D 2022-07-06 18:03:09.645 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /home/fen/.jupyter/jupyter_server_config.json
[D 2022-07-06 18:03:09.645 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_server_config:
        /home/fen/jupyter_server_config.json
[W 2022-07-06 18:03:09.646 SingleUserNotebookApp manager:329]
[D 2022-07-06 18:03:09.737 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/fen/.jupyter/jupyter_notebook_config.json
[D 2022-07-06 18:03:09.737 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /etc/jupyter/jupyter_notebook_config.json
[D 2022-07-06 18:03:09.738 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /usr/local/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-06 18:03:09.738 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /usr/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-06 18:03:09.738 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/fen/.local/etc/jupyter/jupyter_notebook_config.json
[D 2022-07-06 18:03:09.739 SingleUserNotebookApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
        /home/fen/.jupyter/jupyter_notebook_config.json
[W 2022-07-06 18:03:09.739 SingleUserNotebookApp manager:329]
[I 2022-07-06 18:03:09.739 SingleUserNotebookApp manager:340] notebook_shim | extension was successfully linked.
[D 2022-07-06 18:03:09.739 SingleUserNotebookApp serverapp:2009] Raising open file limit: soft 1024->4096; hard 1048576->1048576
[I 2022-07-06 18:03:09.748 SingleUserNotebookApp manager:362] notebook_shim | extension was successfully loaded.
[I 2022-07-06 18:03:09.749 SingleUserNotebookApp mixins:648] Starting jupyterhub-singleuser server version 2.3.1
[I 2022-07-06 18:03:09.750 JupyterHub log:189] 200 GET /hub/api (@127.0.0.1) 0.60ms
[D 2022-07-06 18:03:09.751 SingleUserNotebookApp _version:74] jupyterhub and jupyterhub-singleuser both on version 2.3.1
[I 2022-07-06 18:03:09.751 SingleUserNotebookApp serverapp:2726] Serving notebooks from local directory: /home/fen
[I 2022-07-06 18:03:09.751 SingleUserNotebookApp serverapp:2726] Jupyter Server 1.18.0 is running at:
[I 2022-07-06 18:03:09.751 SingleUserNotebookApp serverapp:2726] http://127.0.0.1:37311/user/fen/tree/
[I 2022-07-06 18:03:09.751 SingleUserNotebookApp serverapp:2726]  or http://127.0.0.1:37311/user/fen/tree/
[I 2022-07-06 18:03:09.751 SingleUserNotebookApp serverapp:2727] Use Control-C to stop this server and shut down all kernels (twice to s
kip confirmation).
[I 2022-07-06 18:03:09.752 SingleUserNotebookApp mixins:597] Updating Hub with activity every 300 seconds
[D 2022-07-06 18:03:09.753 SingleUserNotebookApp mixins:559] Notifying Hub of activity 2022-07-06T22:03:09.743327Z
[D 2022-07-06 18:03:09.774 JupyterHub base:281] Recording first activity for <APIToken('801d...', user='fen', client_id='jupyterhub')>
[D 2022-07-06 18:03:09.785 JupyterHub scopes:301] Authenticated with token <APIToken('801d...', user='fen', client_id='jupyterhub')>
[D 2022-07-06 18:03:09.793 JupyterHub scopes:491] Checking access via scope users:activity
[D 2022-07-06 18:03:09.793 JupyterHub scopes:402] Argument-based access to /hub/api/users/fen/activity via users:activity
[D 2022-07-06 18:03:09.798 JupyterHub users:859] Activity for user fen: 2022-07-06T22:03:09.743327Z
[D 2022-07-06 18:03:09.799 JupyterHub users:877] Activity on server fen/: 2022-07-06T22:03:09.743327Z
[I 2022-07-06 18:03:09.815 JupyterHub log:189] 200 POST /hub/api/users/fen/activity (fen@127.0.0.1) 61.55ms
[I 2022-07-06 18:03:09.867 SingleUserNotebookApp log:189] 302 GET /user/fen/ -> /user/fen/tree/? (@127.0.0.1) 1.23ms
[D 2022-07-06 18:03:09.867 JupyterHub utils:230] Server at http://127.0.0.1:37311/user/fen/ responded with 302
[D 2022-07-06 18:03:09.868 JupyterHub _version:74] jupyterhub and jupyterhub-singleuser both on version 2.3.1
[I 2022-07-06 18:03:09.868 JupyterHub base:963] User fen took 0.752 seconds to start
[I 2022-07-06 18:03:09.868 JupyterHub proxy:286] Adding user fen to proxy /user/fen/ => http://127.0.0.1:37311
[D 2022-07-06 18:03:09.869 JupyterHub proxy:821] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/fen
18:03:09.873 [ConfigProxy] info: Adding route /user/fen -> http://127.0.0.1:37311
18:03:09.874 [ConfigProxy] info: Route added /user/fen -> http://127.0.0.1:37311
18:03:09.875 [ConfigProxy] info: 201 POST /api/routes/user/fen
[I 2022-07-06 18:03:09.876 JupyterHub log:189] 302 GET /hub/spawn/fen?next=%2Fhub%2Fuser%2Ffen%2Ftree -> /hub/spawn-pending/fen?next=%2F
hub%2Fuser%2Ffen%2Ftree (fen@::ffff:10.9.1.1) 761.51ms
[D 2022-07-06 18:03:09.892 JupyterHub scopes:491] Checking access via scope servers
[D 2022-07-06 18:03:09.892 JupyterHub scopes:402] Argument-based access to /hub/spawn-pending/fen via servers
[I 2022-07-06 18:03:09.892 JupyterHub log:189] 302 GET /hub/spawn-pending/fen?next=%2Fhub%2Fuser%2Ffen%2Ftree -> /hub/user/fen/tree (fen
@::ffff:10.9.1.1) 1.78ms
[D 2022-07-06 18:03:09.912 JupyterHub scopes:491] Checking access via scope access:servers
[D 2022-07-06 18:03:09.912 JupyterHub scopes:402] Argument-based access to /hub/user/fen/tree via access:servers
[I 2022-07-06 18:03:09.913 JupyterHub log:189] 302 GET /hub/user/fen/tree -> /user/fen/tree?redirects=1 (fen@::ffff:10.9.1.1) 1.85ms
[D 2022-07-06 18:03:09.978 SingleUserNotebookApp auth:584] No user identified
[D 2022-07-06 18:03:09.979 SingleUserNotebookApp handlers:273] Using contents: services/contents
[D 2022-07-06 18:03:10.020 SingleUserNotebookApp handlers:881] Path favicon.ico served from /home/fen/.local/lib/python3.10/site-package
s/jupyter_server/static/favicon.ico
[D 2022-07-06 18:03:10.020 SingleUserNotebookApp handlers:881] Path style/bootstrap.min.css served from /home/fen/.local/lib/python3.10/
site-packages/jupyter_server/static/style/bootstrap.min.css
[D 2022-07-06 18:03:10.020 SingleUserNotebookApp handlers:881] Path style/bootstrap-theme.min.css served from /home/fen/.local/lib/pytho
n3.10/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
[D 2022-07-06 18:03:10.021 SingleUserNotebookApp handlers:881] Path style/index.css served from /home/fen/.local/lib/python3.10/site-pac
kages/jupyter_server/static/style/index.css
[W 2022-07-06 18:03:10.021 SingleUserNotebookApp log:189] 404 GET /user/fen/tree?redirects=1 (@::ffff:10.9.1.1) 44.35ms

My pip list output is also below:

(sparkfly) fen@fenix:~$ pip list
Package              Version
-------------------- -----------
alembic              1.8.0
anyio                3.6.1
argon2-cffi          21.3.0
argon2-cffi-bindings 21.2.0
asttokens            2.0.5
async-generator      1.10
attrs                21.4.0
Babel                2.10.3
backcall             0.2.0
beautifulsoup4       4.11.1
bleach               5.0.1
certifi              2022.6.15
certipy              0.1.3
cffi                 1.15.1
charset-normalizer   2.1.0
cryptography         37.0.2
debugpy              1.6.0
decorator            5.1.1
defusedxml           0.7.1
entrypoints          0.4
executing            0.8.3
fastjsonschema       2.15.3
greenlet             1.1.2
idna                 3.3
ipykernel            6.15.0
ipython              8.4.0
ipython-genutils     0.2.0
jedi                 0.18.1
Jinja2               3.1.2
json5                0.9.8
jsonschema           4.6.1
jupyter-client       7.3.4
jupyter-core         4.10.0
jupyter-server       1.18.0
jupyter-telemetry    0.1.0
jupyterhub           2.3.1
jupyterlab           3.4.3
jupyterlab-pygments  0.2.2
jupyterlab-server    2.14.0
Mako                 1.2.1
MarkupSafe           2.1.1
matplotlib-inline    0.1.3
mistune              0.8.4
nbclassic            0.4.0
nbclient             0.6.6
nbconvert            6.5.0
nbformat             5.4.0
nest-asyncio         1.5.5
notebook             6.4.12
notebook-shim        0.1.0
oauthlib             3.2.0
packaging            21.3
pamela               1.0.0
pandocfilters        1.5.0
parso                0.8.3
pexpect              4.8.0
pickleshare          0.7.5
pip                  22.0.2
prometheus-client    0.14.1
prompt-toolkit       3.0.30
psutil               5.9.1
ptyprocess           0.7.0
pure-eval            0.2.2
pycparser            2.21
Pygments             2.12.0
pyOpenSSL            22.0.0
pyparsing            3.0.9
pyrsistent           0.18.1
python-dateutil      2.8.2
python-json-logger   2.0.2
pytz                 2022.1
pyzmq                23.2.0
requests             2.28.1
ruamel.yaml          0.17.21
ruamel.yaml.clib     0.2.6
Send2Trash           1.8.0
setuptools           59.6.0
six                  1.16.0
sniffio              1.2.0
soupsieve            2.3.2.post1
SQLAlchemy           1.4.39
stack-data           0.3.0
terminado            0.15.0
tinycss2             1.1.1
tornado              6.2
traitlets            5.3.0
urllib3              1.26.9
wcwidth              0.2.5
webencodings         0.5.1
websocket-client     1.3.3

Im hitting on the same with our redeployment with latest version:
JupyterHub 2.3.1 20220707102716

When using version: 2.3.0 it works.

Could you show us your config please (with secrets redacted)? Thanks!