Non-admin users can't log in because of token permission errors

Hi everyone,

I am building Jupyterhub on Kubernetes and I noticed that users that are not allowed as admins trigger the following errors:
“ValueError: Requested token role server of <APIToken(’…’, user=’*****’, client_id=‘jupyterhub’)> has more permissions than the token owner”

“Token requesting scopes exceeding owner *****: {‘users:activity’, ‘access:servers’, ‘read:users:activity’}”

If the user is then allowed as admin then the error will not appear. I haven’t done any changes to the roles.py file. Has anyone had this error and found a fix for it? Many thanks!

Hi! Can you show us your full configuration with secrets redacted, and tell us your Z2JH version?

Hello, sorry for the late reply!
For jupyterhub i am using the base image k8s-hub at version 1.1.3. Over it I install the latest version of Jupyterhub from github, on which I added additional features.
The only changes I have done to the source code in jupyterhub were adding an additional column for the groups table in the database. The scopes and roles have not been changed and the normal jupyterhub (without k8s) deployment works well.

Thanks for testing 2.0!

JupyterHub 2.0 is a major release, so you’ll probably need to use the latest dev version of the chart if you want to also use the latest dev version of jupyterhub itself.

Can you share your configuration of users and roles (with usernames obfuscated, if needed)? Have you customized either the user role or the server role?

The error means that a token was assigned a role (the default server role) which contains scopes the user itself does not have. This could mean an overlapping scope where the user has a filter (e.g. users:activity!user) while the server role has the full users:activity, which would mean permissions for all users, which would be forbidden.

You would see this error if you declared one or both of the user and server roles such that the server role exceeded the user role. This could be done by either increasing the server role (default: ['users:activity!user', 'access:servers!user']) or decreasing the user role: (default: ['self']).

If you don’t have any role-related configuration, my hunch is that something here prevented JupyterHub’s default role assignment for existing users. I don’t know if it was a bug in the JupyterHub 2.0 beta or in the customizations you’ve made. First, make sure your user having trouble actually has the user role.

It would be useful to check the database status.

One way to do this is to poke around with jupyterhub.dbutil. To launch the db shell in your hub pod:

hub=$(kubectl get pod --no-headers -l component=hub | awk '{print $1}')
kubectl exec -it ${hub} -- bash

Then in the shell, run:

python3 -m pip install ipython
python3 -m jupyterhub.dbutil shell

At which point, you’ll have an IPython shell connected to your JupyterHub database.

You can then collect info about the user and roles:

from jupyterhub import scopes
# look up the user
user = orm.User.find(db, name="username")

# check user's role assignments
print(user.roles)

# check user's expanded roles
print(scopes.get_scopes_for(user))

# check server role
server_role = orm.Role.find(db, name="server")
print(server_role.scopes)

Thank you very much for your answer!
I didn’t do any changes to the roles, so the default values have been left untouched.
The IPython code helped me find the problem. It seems that users that sign in and are not added as admins in the c.Authenticator.admin_users do not get any role at all. I will try to force assign the user role to non-admin accounts and see if it fixes the problem.
Update:
Solved by adding the user role manually using the roles.grant_role() function in the pre_auth_hook. Many thanks!

What Authenticator? Users should be granted the default ‘user’ role on login, but I will double check tomorrow.

I am using the OAuthenticator with keycloak.

And not at all customized? When a new user is created, they are assigned the default ‘user’ role. I’ve just added a test to verify this, and it is indeed the case.

If the user was somehow created without role assignment, they won’t get re-assigned the role on subsequent logins (i.e. preserving revocations). It would be useful to understand how this could happen, though. Do you see it consistently with any new login?

Yes the user roles now remain consistent after assigning them to the users. It could be possible that the problem was caused by my authenticator which has been a bit altered (no changes to roles, however).

Before fixing it, it was interesting that only admin roles were assigned if the users were allowed in config.py. If afterwards I would remove a user from allowed_admin, they would remain without any role in the database.

I will also test it further and see if I can find where the bug was caused.

Ah, that’s a lead!Were the users affected previously admins, but then lost admin status? I think there’s a sequence where that could result in not gaining user status.

I’ll try to write up a test and issue on GitHub.

Hi @minrk ,
I think the problem is here:
When the hub starts, we have to add the users in db to the authenticator. And adding them using the method add_user from the authenticator

where we should use another way to see if we should add one user or not (or maybe to have an allowed_all flag). Because an empty set means that we allow all authenticated users.

Does it make sense? I remove the if at that line then the problem was solved for me.