Admin Panel empty, getting 403 '_xsrf' argument missing from GET

JupyterHub
1

Hi there,

I upgraded from JupyterHub 3 to 4.0.2 today, created a new database, and sadly I can’t access the admin panel anymore, or better said it is empty.

I of course added the user (muellerl) to the c.Authenticator.admin_users dict. While trying around I also created a role + service and specified my user in this role.

I’m using a remote header to authenticate: https://github.com/cwaldbieser/jhub_remote_user_authenticator/blob/fb5a7ec8c1b567917b4dabab1d9167e499ea2c8a/jhub_remote_user_authenticator/remote_user_auth.py

The logs tell:

Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [I 2023-09-26 19:46:50.478 JupyterHub log:191] 302 GET /hub/login?next=%2Fhub%2Fadmin -> /hub/admin (@127.0.0.1) 10.35ms
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.500 JupyterHub base:299] Recording first activity for <User(muellerl 0/1 running)>
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.505 JupyterHub base:344] Refreshing auth for muellerl
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.508 JupyterHub scopes:877] Checking access to /hub/admin via scope admin-ui
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.508 JupyterHub scopes:677] Unrestricted access to /hub/admin via admin-ui
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.523 JupyterHub user:431] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for muellerl:
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:50.524 JupyterHub systemdspawner:155] user:muellerl Initialized spawner with unit jupyter-muellerl-singleuser
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [I 2023-09-26 19:46:50.529 JupyterHub log:191] 200 GET /hub/admin (muellerl@127.0.0.1) 30.03ms
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [I 2023-09-26 19:46:50.788 JupyterHub log:191] 302 GET /user/muellerl/api/events/subscribe?token=[secret] -> /hub/user/muellerl/api/events/subscribe?token=[secret] (@127.0.0.1) 0.64ms
Sep 26 19:46:50 jupyterhub jupyterhub[244487]: [I 2023-09-26 19:46:50.992 JupyterHub log:191] 302 GET /logo -> /hub/logo (@127.0.0.1) 0.48ms
Sep 26 19:46:51 jupyterhub jupyterhub[244487]: [D 2023-09-26 19:46:51.026 JupyterHub log:191] 304 GET /hub/logo (@127.0.0.1) 2.78ms
Sep 26 19:46:51 jupyterhub jupyterhub[244487]: [W 2023-09-26 19:46:51.040 JupyterHub web:1869] 403 GET /hub/api/users?include_stopped_servers&offset=0&limit=50&name_filter= (127.0.0.1): '_xsrf' argument missing from GET
Sep 26 19:46:51 jupyterhub jupyterhub[244487]: [W 2023-09-26 19:46:51.040 JupyterHub log:191] 403 GET /hub/api/users?include_stopped_servers&offset=0&limit=50&name_filter= (muellerl@127.0.0.1) 1.69ms

Does someone have an idea?

edit:
The tokens panel also does show API request failed (403): '_xsrf' argument missing from POST when trying to create a new token.

2

To rule out jhub_remote_user_authenticator, can you test one of the standard authenticators (e.g. the dummy authenticator), and see if the problem still occurs?

3

I can try that tomorrow, however, the authenticator was in use since Hub version 2. Did something change in version 4?

edit: Using the DummyAuthenticator does not solve the problem.
edit2: I’m trying to narrow it down, the error comes from the tonardo server check_xsrf_cookie function. There is no header set to compare to, only the cookie.
Maybe I’m missing some options in Apache2? I also upgraded to apache 2.4 and am running behind a reverse proxy.

Can someone with knowledge about the cross site checking help maybe?

edit3: okay I did narrow it down:
I was overriding / providing an old custom page.html template. The newer templates do provide the xsrf_token in the jhdata objects which is populated by jinja / when rendering the site.
Of course I did not have this attribute in my old overriding page.html, which was causing the error. Therefore deleting my custom page.html solved the issue…

Maybe note this in an upgrade guide if someone out there is also overriding the page.html?

Best regards

2 Likes
4

Hi,

How did you fix the issue?

best

5

Hi bpfrd,

I did narrow it down to the fact that the xsrf token was missing for the requests.
As I described above, I had an old custom page.html that was overriding the newer page.html (from updates).
My custom page.html lived in /etc/jupyterhub somewhere. After deleting this custom page.html the issue was gone.

Best regards

1 Like