404 on Jupyterhub (2.3.1)

JupyterHub
1

Hi.

We have a nicely running 1.5.0 installation, but I had a desire to upgrade (to 2.3.1). The configuration in currently apache ( SSL termination + Kerberos auth) - systemdSpawner + headerauthenticator. - it works on 1.5.0 but when upgrading it launches the “jupyterhub-singleuser” but when accessing it gives a 404 on /user//tree

Jun 14 21:00:11 tdkcphapp135rd systemd[1]: Started /bin/bash -c cd /home/jskr && exec jupyterhub-singleuser --debug.
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:12.372 SingleUserNotebookApp application:183] Searching [’/net/dkcphapp003rd/home/jskr’, ‘/net/dkcphapp003rd/home/jskr/.jupyter’, ‘/net/dkcphapp003rd/home/virtualenvs/20220531-161303/etc/jupyter’, ‘/usr/local/etc/jupyter’, ‘/etc/jupyter’] for config files
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:12.374 SingleUserNotebookApp application:833] Looking for jupyter_config in /net/dkcphapp003rd/home/topsoe/jskr/.jupyter
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:12.374 SingleUserNotebookApp application:833] Looking for jupyter_config in /net/dkcphapp003rd/home/jskr
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:12.375 SingleUserNotebookApp application:833] Looking for jupyter_server_config in /net/dkcphapp003rd/home/jskr/.jupyter
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:12.375 SingleUserNotebookApp application:833] Looking for jupyter_server_config in /net/dkcphapp003rd/home/jskr
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: #011/net/dkcphapp003rd/home/topsoe/jskr/.jupyter/jupyter_server_config.json
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: #011/net/dkcphapp003rd/home/topsoe/jskr/jupyter_server_config.json
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:12.885 SingleUserNotebookApp serverapp:2671] Serving notebooks from local directory: /home//jskr
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:12.885 SingleUserNotebookApp serverapp:2671] http://127.0.0.1:39397/user/jskr/tree/
Jun 14 21:00:12 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:12.885 SingleUserNotebookApp serverapp:2671] or http://127.0.0.1:39397/user/jskr/tree/
Jun 14 21:00:14 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:14.583 SingleUserNotebookApp log:189] 302 GET /user/jskr/ → /user/jskr/tree/? (@127.0.0.1) 1.22ms
Jun 14 21:00:15 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:15.287 SingleUserNotebookApp log:189] 302 GET /user/jskr/ → /user/jskr/tree/? (@127.0.0.1) 1.27ms
Jun 14 21:00:15 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:15.831 SingleUserNotebookApp log:189] 302 GET /user/jskr/tree/ → /user/jskr/tree (@127.0.0.1) 1.28ms
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [W 2022-06-14 21:00:16.145 SingleUserNotebookApp log:189] 404 GET /user/jskr/tree (@127.0.0.1) 29.39ms

From the systemd log i get:
Jun 14 21:00:15 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:15.287 SingleUserNotebookApp log:189] 302 GET /user/jskr/ → /user/jskr/tree/? (@127.0.0.1>
Jun 14 21:00:15 tdkcphapp135rd bash[1257780]: [I 2022-06-14 21:00:15.831 SingleUserNotebookApp log:189] 302 GET /user/jskr/tree/ → /user/jskr/tree (@127.0.>
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.116 SingleUserNotebookApp auth:584] No user identified
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.117 SingleUserNotebookApp handlers:257] Using contents: services/contents
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.141 SingleUserNotebookApp handlers:857] Path favicon.ico served from /net/dkcphapp003rd>
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.142 SingleUserNotebookApp handlers:857] Path style/bootstrap.min.css served from /net/d>
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.143 SingleUserNotebookApp handlers:857] Path style/bootstrap-theme.min.css served from >
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:00:16.144 SingleUserNotebookApp handlers:857] Path style/index.css served from /net/dkcphapp0>
Jun 14 21:00:16 tdkcphapp135rd bash[1257780]: [W 2022-06-14 21:00:16.145 SingleUserNotebookApp log:189] 404 GET /user/jskr/tree (@127.0.0.1) 29.39ms
Jun 14 21:05:30 tdkcphapp135rd bash[1257780]: [D 2022-06-14 21:05:30.659 SingleUserNotebookApp mixins:559] Notifying Hub of activity 2022-06-14T21:00:12.

Especially the “No user identified” - gives me some suspicion.

Good advice to debug?

Thanks alot

2

Can you share the env (conda list or pip freeze)? My hunch is that you ended up with Jupyter Server, but not JupyterLab or nbclassic. A pip install nbclassic or pip install jupyterlab ought to work.

There’s still something not quite right in how jupyterhub 2 picks the single-user environment in the transition from notebook to jupyter server, because jupyter server is now the default, but it’s possible for jupyter server to run without any UI available.