JupyterHub on K8s: 403 Forbidden for Users Added via Admin Page but not the Helm Chart Config

JupyterHub Zero to JupyterHub on Kubernetes
,
1

We have a JupyterHub for K8s running on top of an Openstack cloud (NFS’s Jetstream2). One of our admin users recently contacted us that his students were getting a 403 Forbidden message when attempting to log in, even after he added their usernames via the Admin panel, <hub-url>/hub/admin. We are using the GitHub OAuth authentication mechanism, something that we’re well acquainted with.

After some minor troubleshooting, we realized that, if we add users via the helm chart config.yaml and reinstall the new release, that users no longer get the error 403 message and get redirected to their singleuser JupyterLab server.

As users are able to log in when added via config.yaml, I suspect this is a JupyterHub related issue, not a GitHub OAuth issue.

A separate JupyterHub with a similar config displays the same behavior. They are both running JupyterHub 4.0.2. E.g.:

$ kubectl exec -n jhub <hub-pod> -- jupyterhub --version
4.0.2

We are using helm chart version 3.0.3.

I’ve searched both the jupyterhub/jupyterhub GitHub issues page and these forums for anybody that was experiencing a similar issue, to no avail. Does anybody else have this problem, or know of a cause/fix?

Thanks for reading!

– ana v. e.

(configs and logs below)

Here’s the relevant part of the config, with identifying and irrelevant information redacted:

hub:
  config:
    Authenticator:
      admin_users:
        - "admins"
      allowed_users:
        - "users"

    GitHubOAuthenticator:
      client_id: "xxx"
      client_secret: "xxx"
      oauth_callback_url: "<jhub-url>:443/oauth_callback"
    JupyterHub:
      authenticator_class: github

I decided to set debug.enabled: true to see if I could gather any useful information from the JHub logs, but I don’t see anything past what you’d expect for an unauthorized user.

Here are the logs from when I add a user via the Admin page showing that the user has been added:

[D 2024-06-14 17:33:18.768 JupyterHub scopes:877] Checking access to /hub/api/users via scope admin:users
[D 2024-06-14 17:33:18.768 JupyterHub scopes:677] Unrestricted access to /hub/api/users via admin:users
[D 2024-06-14 17:33:18.769 JupyterHub roles:281] Assigning default role to User <user>
[I 2024-06-14 17:33:18.771 JupyterHub roles:238] Adding role user for User: <user>
[D 2024-06-14 17:33:18.794 JupyterHub roles:281] Assigning default role to User <user>

When attempting a login with that user:

[I 2024-06-14 17:38:57.449 JupyterHub log:191] 200 GET /hub/login (@<IP>) 1.62ms
[D 2024-06-14 17:38:58.671 JupyterHub log:191] 200 GET /hub/static/favicon.ico?v=<string> (@<IP>) 0.66ms
[D 2024-06-14 17:38:58.937 JupyterHub reflector:362] events watcher timeout
[D 2024-06-14 17:38:58.937 JupyterHub reflector:281] Connecting events watcher
[D 2024-06-14 17:38:59.078 JupyterHub log:191] 200 GET /hub/health (@<clusterIP>) 0.67ms
[I 2024-06-14 17:38:59.744 JupyterHub oauth2:102] OAuth redirect: <hub-url>:443/oauth_callback
[D 2024-06-14 17:38:59.744 JupyterHub base:587] Setting cookie oauthenticator-state: {'httponly': True, 'secure': True, 'expires_days': 1}
[I 2024-06-14 17:38:59.745 JupyterHub log:191] 302 GET /hub/oauth_login?next= -> https://github.com/login/oauth/authorize?response_type=code&redirect_uri=<URI>&client_id=<OAuth-cli-ID>&state=[secret] (@<IP>) 1.18ms
[I 2024-06-14 17:39:00.630 JupyterHub log:191] 302 GET /oauth_callback?code=[secret]&state=[secret] -> /hub/oauth_callback?code=[secret]&state=[secret] (@<IP>) 0.69ms
[D 2024-06-14 17:39:01.079 JupyterHub log:191] 200 GET /hub/health (@<clusterIP>) 0.78ms
[D 2024-06-14 17:39:01.079 JupyterHub log:191] 200 GET /hub/health (@<clusterIP>) 0.75ms
[W 2024-06-14 17:39:01.182 JupyterHub auth:533] User '<user>' not allowed.
[W 2024-06-14 17:39:01.182 JupyterHub base:843] Failed login for unknown user
[W 2024-06-14 17:39:01.182 JupyterHub web:1869] 403 GET /hub/oauth_callback?code=<code> (<IP>): Sorry, you are not currently authorized to use this hub. Please contact the hub administrator.
[D 2024-06-14 17:39:01.182 JupyterHub base:1371] No template for 403
[W 2024-06-14 17:39:01.183 JupyterHub log:191] 403 GET /hub/oauth_callback?code=[secret]&state=[secret] (@<IP>) 448.46ms
2

With OAuthenticator, you will have to explicitly set a list of users in allow_users in JupyterHub config. I dont think adding a new user from Hub’s admin page will add that user in allow_users list automatically. There is a allow_all config parameter when set to True, you will be able to add new users from Hub’s admin page and authenticator will allow that user.

3

Hello,

Thanks for the response.

I can confirm with multiple years of experience that admin users do have the ability to add new users via the admin page, even without changing the config via the helm chart, and without the allow_all parameter being set to true. See below for “proof”.

In fact, the user that reported this issue has been working with us in this manner for multiple years as well.

Additionally, we do not want it to be the case that anybody with a valid GitHub account be able to access this JupyterHub.

– ana v. e.

Here are the config and the logs for when I add a user via an admin panel and login with that user successfully on JupyterHub 3.0.0:

$ kubectl exec -n jhub <jhub-pod> -- jupyterhub --version
3.0.0

Config, once again with identifying and irrelevant information redacted. Let it be noted that the <user> referred to in the logs below is not in the allowed_users list defined here:

hub:
  config:
    Authenticator:
      admin_users:
        - "admins"
      #If you have a large list of users, consider using allowed_users.yaml
      allowed_users:
        - "users"
    GitHubOAuthenticator:
      client_id: "xxx"
      client_secret: "xxx"
      oauth_callback_url: "<jhub-url>:443/oauth_callback"
    JupyterHub:
      authenticator_class: github

Logs (non-verbose/non-debug) when adding new user via admin panel:

[I 2024-06-15 14:52:40.123 JupyterHub roles:238] Adding role user for User: <user>
...
[I 2024-06-15 14:53:00.825 JupyterHub base:810] User logged in: <user>
[I 2024-06-15 14:53:00.826 JupyterHub log:186] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/ (@<IP>) 359.15ms
[I 2024-06-15 14:53:00.890 JupyterHub log:186] 302 GET /hub/ -> /hub/spawn (<user>@<IP>) 16.76ms
[I 2024-06-15 14:53:00.950 JupyterHub log:186] 200 GET /hub/spawn (<user>@<IP>) 12.45ms
[I 2024-06-15 14:53:03.731 JupyterHub provider:651] Creating oauth client jupyterhub-user-<user>
[I 2024-06-15 14:53:03.763 JupyterHub spawner:2509] Attempting to create pvc claim-<user>, with timeout 3
[I 2024-06-15 14:53:03.765 JupyterHub log:186] 302 POST /hub/spawn -> /hub/spawn-pending/<user> (<user>@<IP>) 62.70ms
[I 2024-06-15 14:53:03.785 JupyterHub spawner:2469] Attempting to create pod jupyter-<user>, with timeout 3
[I 2024-06-15 14:53:03.815 JupyterHub pages:394] <user> is pending spawn
[I 2024-06-15 14:53:03.818 JupyterHub log:186] 200 GET /hub/spawn-pending/<user> (<user>@<IP>) 4.75ms
[I 2024-06-15 14:53:16.866 JupyterHub log:186] 200 GET /hub/api (@10.233.68.162) 0.66ms
[I 2024-06-15 14:53:17.033 JupyterHub log:186] 200 POST /hub/api/users/<user>/activity (<user>@<clusterIP>) 40.14ms
[W 2024-06-15 14:53:17.108 JupyterHub _version:37] Single-user server has no version header, which means it is likely < 0.8. Expected 3.0.0
[I 2024-06-15 14:53:17.108 JupyterHub base:963] User <user> took 13.404 seconds to start
[I 2024-06-15 14:53:17.109 JupyterHub proxy:333] Adding user <user> to proxy /user/<user>/ => http://<clusterIP>:8888
[I 2024-06-15 14:53:17.111 JupyterHub users:749] Server <user> is ready
4

Check out allow_existing_users option in oauthenticator and jupyterhub 5’s base Authenticator class.

https://oauthenticator.readthedocs.io/en/latest/reference/api/gen/oauthenticator.oauth2.html#oauthenticator.oauth2.OAuthenticator.allow_existing_users

In oauthenticator 16 and jupyterhub 5, the behavior changed, where this new config makes the desired behavior of explicit.

2 Likes
5

Thanks Erik, as you say, adding this option to our config results in the exact behavior that I’m familiar with. To make things explicit for anybody that’s also peaking this thread, my config now looks like:

hub:
  config:
    Authenticator:
      admin_users:
        - "admins"
      #If you have a large list of users, consider using allowed_users.yaml
      allowed_users:
        - "users"
      allow_existing_users: true
    GitHubOAuthenticator:
      client_id: "xxx"
      client_secret: "xxx"
      oauth_callback_url: "<jhub-url>:443/oauth_callback"
    JupyterHub:
      authenticator_class: github
1 Like