Non-admin users can't log in because of token permission errors

JupyterHub Zero to JupyterHub on Kubernetes
,
1

Hi everyone,

I am building Jupyterhub on Kubernetes and I noticed that users that are not allowed as admins trigger the following errors:
“ValueError: Requested token role server of <APIToken(’…’, user=’*****’, client_id=‘jupyterhub’)> has more permissions than the token owner”

“Token requesting scopes exceeding owner *****: {‘users:activity’, ‘access:servers’, ‘read:users:activity’}”

If the user is then allowed as admin then the error will not appear. I haven’t done any changes to the roles.py file. Has anyone had this error and found a fix for it? Many thanks!

2

Hi! Can you show us your full configuration with secrets redacted, and tell us your Z2JH version?

3

Hello, sorry for the late reply!
For jupyterhub i am using the base image k8s-hub at version 1.1.3. Over it I install the latest version of Jupyterhub from github, on which I added additional features.
The only changes I have done to the source code in jupyterhub were adding an additional column for the groups table in the database. The scopes and roles have not been changed and the normal jupyterhub (without k8s) deployment works well.

4

Thanks for testing 2.0!

JupyterHub 2.0 is a major release, so you’ll probably need to use the latest dev version of the chart if you want to also use the latest dev version of jupyterhub itself.

Can you share your configuration of users and roles (with usernames obfuscated, if needed)? Have you customized either the user role or the server role?

The error means that a token was assigned a role (the default server role) which contains scopes the user itself does not have. This could mean an overlapping scope where the user has a filter (e.g. users:activity!user) while the server role has the full users:activity, which would mean permissions for all users, which would be forbidden.

You would see this error if you declared one or both of the user and server roles such that the server role exceeded the user role. This could be done by either increasing the server role (default: ['users:activity!user', 'access:servers!user']) or decreasing the user role: (default: ['self']).

If you don’t have any role-related configuration, my hunch is that something here prevented JupyterHub’s default role assignment for existing users. I don’t know if it was a bug in the JupyterHub 2.0 beta or in the customizations you’ve made. First, make sure your user having trouble actually has the user role.

It would be useful to check the database status.

One way to do this is to poke around with jupyterhub.dbutil. To launch the db shell in your hub pod:

hub=$(kubectl get pod --no-headers -l component=hub | awk '{print $1}')
kubectl exec -it ${hub} -- bash

Then in the shell, run:

python3 -m pip install ipython
python3 -m jupyterhub.dbutil shell

At which point, you’ll have an IPython shell connected to your JupyterHub database.

You can then collect info about the user and roles:

from jupyterhub import scopes
# look up the user
user = orm.User.find(db, name="username")

# check user's role assignments
print(user.roles)

# check user's expanded roles
print(scopes.get_scopes_for(user))

# check server role
server_role = orm.Role.find(db, name="server")
print(server_role.scopes)
5

Thank you very much for your answer!
I didn’t do any changes to the roles, so the default values have been left untouched.
The IPython code helped me find the problem. It seems that users that sign in and are not added as admins in the c.Authenticator.admin_users do not get any role at all. I will try to force assign the user role to non-admin accounts and see if it fixes the problem.
Update:
Solved by adding the user role manually using the roles.grant_role() function in the pre_auth_hook. Many thanks!

6

What Authenticator? Users should be granted the default ‘user’ role on login, but I will double check tomorrow.

7

I am using the OAuthenticator with keycloak.

8

And not at all customized? When a new user is created, they are assigned the default ‘user’ role. I’ve just added a test to verify this, and it is indeed the case.

If the user was somehow created without role assignment, they won’t get re-assigned the role on subsequent logins (i.e. preserving revocations). It would be useful to understand how this could happen, though. Do you see it consistently with any new login?

9

Yes the user roles now remain consistent after assigning them to the users. It could be possible that the problem was caused by my authenticator which has been a bit altered (no changes to roles, however).

Before fixing it, it was interesting that only admin roles were assigned if the users were allowed in config.py. If afterwards I would remove a user from allowed_admin, they would remain without any role in the database.

I will also test it further and see if I can find where the bug was caused.

10

Ah, that’s a lead!Were the users affected previously admins, but then lost admin status? I think there’s a sequence where that could result in not gaining user status.

I’ll try to write up a test and issue on GitHub.

11

Hi @minrk ,
I think the problem is here:
When the hub starts, we have to add the users in db to the authenticator. And adding them using the method add_user from the authenticator

where we should use another way to see if we should add one user or not (or maybe to have an allowed_all flag). Because an empty set means that we allow all authenticated users.

Does it make sense? I remove the if at that line then the problem was solved for me.

12

I am experiencing a similar issue. I have Jupyterhub 3.0.0 deployed on a linux server and we are spawning user processes with jupyterhub-singleuser. I am using the GenericOAuthenticator to authenticate users via keycloak using the c.GenericOAuthenticator.allowed_groups = [‘CN=GROUPNAME,ou=groups,ou=Customer,ou=Costomerorganization,o=CustomerName’]

For admins, I am using the c.GenericOAuthenticator.admin_groups as well.

I believe the problem is that the non-admin users who are in an approved group are not being allocated any user privileges/roles. I can see in the Hub Control panel, admin users have

0 | user
1 | admin

And for non-admin users there is only the 0 | user entry. When the user hits the external service, I do see the following scopes being passed access:servers!user=username, read:users:activity!user=username and users:activity!user=username to the external service. Users are getting a 403 error but the admins are allowed to access the service because admin: True I believe.

c.JupyterHub.service = [
{
‘name’: ‘external-service’,
‘oauth_client_id’: ‘service-external-client’,
‘api_token’: ‘token’
‘oauth_redirect_uri’: ‘https://servername/callback’,
}
]

c.JupyterHub.load_roles = [
{
‘name’: ‘nonadminUsers’,
‘scopes’: [
‘users:activitiy!user’,
‘access:services!user’;
‘access:servers!user’,
],
‘services’: [
‘service-external-client’,
],
}
]

I am pretty sure this is self inflicted as I have not set c.GenericOAuthenticator.allowed_users or implemented the roles.grant_role() function (mainly because I am not sure of where to put it :slight_smile: )

13

Wanted to add the last entries I say in the jupyterhub.log file

Validating redirection uri "https://server.domain/jupyterhub/callback for client service-external-client

Validating access scopes for client ‘service-external-client’(<OAuthClient(identifier='service-external-client)>)

Allowing request for role(s) for service-external-client:

Then failes with User (usrname) not allowed to access jupyterHub service ‘external-service’

Not sure what both scopes and roles are not defined, or at least I think means empty

14

Ok, after rereading the thread (several times) I have determined that my problem was the scope definitions.
c.JupyterHub.service = [
{
‘name’: ‘external-service’,
‘oauth_client_id’: ‘service-external-client’,
‘api_token’: ‘tokenpassedin’
‘oauth_redirect_uri’: ‘https://servername/callback’,
}
c.JupyterHub.load_roles = [
{
‘name’: ‘nonadminUsers’,
‘scopes’: [
‘list:users’,
'read:users:activity’,
‘access:services;
‘access:servers’,
‘access:services!service=external-service’
],
‘services’: [
‘external-service’,
],
‘groups’: [‘token’],
}

The above configuration worked.

Also thinking my reference to no user role above via the “0 | user” was also a mistake. The ‘0’ is the role identifier I believe. Sorry about that, still very new to Jupyterhub.

Is there a way to automatically add users to the ‘token’ group during singleuser launch?

15

There’s a default user role that all users are assigned to. You could try overriding it?
https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html

16

@manics Thanks for the response. I have read that page and I am not sure exactly how to override that unless I edit roles.py??

17

@manics Sorry, I was not more specific there. Yes, I did see the section on how to override roles by using the scopes ‘inherit’ and that did not solve the problem for me. I am thinking the only way I can add the user to the group “token” upon singleuser spawn is to hit the REST API call POST /groups/token/users. I will try adding it to the pre-spawn shell

18

Just wanted to report that adding the REST PUT call to add a user to a group worked. I followed the recommendations in the Using JupyterHub’s REST API — JupyterHub 3.0.0 documentation

Thanks for the help, most appreciated.

1 Like