Admin page error after upgrading to 2.0.1

JupyterHub
,
1

I am unable to list any users in the admin page after upgrading from 2.0.0 to 2.0.1. I have confirmed that I have admin permissions in the users table and in the use_role_map table.

The error is
Not authorizing access to /hub/api/users. Requires any of [list:users], not derived from scopes []

It works when I downgrade to 2.0.0.

2

Hi! Could you show us your full JupyterHub configuration with secrets redacted? Thanks!

3

Created a minimum reproducible example

config

from jupyter_client.localinterfaces import public_ips

c.JupyterHub.hub_ip = public_ips()[0]  
c.JupyterHub.allow_named_servers = True

# Authenticator 
c.JupyterHub.authenticator_class = 'my_module.CustomAuthenticator'

c.CustomAuthenticator.enable_auth_state = True
c.CustomAuthenticator.admin_users = {'gramesh'}

Authenticator

from jupyterhub.auth import Authenticator

class CustomAuthenticator(Authenticator):

    async def authenticate(self, handler, data):
        username = data['username']
        pwd = data['password']

        return {
            'name': username,

        }

    async def refresh_user(self, handler=None, force=False):
        return False
4

This bit:

async def refresh_user(self, handler=None, force=False):
        return False

means that after Authenticator.auth_refresh_age (default: 5 minutes), a user will be unconditionally logged out.

I don’t believe that 2.0.1 is relevant to this, but rather the timing of the tests - if you visit the admin page before your auth expires, and then attempt to take another action after it expires, you will see an error like this.

Here is a complete minimal repro config:


from jupyterhub.auth import Authenticator

class CustomAuthenticator(Authenticator):

    async def authenticate(self, handler, data):
        username = data['username']
        pwd = data['password']

        return {
            'name': username,
        }

    async def refresh_user(self, user, handler=None):
        self.log.info(f"Revoking expired authentication for user {user.name}")
        return False

c = get_config()  # noqa

c.JupyterHub.authenticator_class = CustomAuthenticator

c.CustomAuthenticator.admin_users = {'gramesh'}
# always trigger refresh on spawn
c.CustomAuthenticator.refresh_pre_spawn = True
# expire auth _really_ fast so we can easily test
c.CustomAuthenticator.auth_refresh_age = 10

# test Spawner
c.JupyterHub.spawner_class = "simple"

So if there is a bug, it is probably handling the rejected request to prompt login again (as is the intended result of refresh_user returning False), rather than 403. This should already be the case for most requests, but there are certainly more places it can come up.

So here’s perhaps the key question: why do you have refresh_user returning False? This will likely mean permission errors every 5 minutes.

5

Thanks @minrk,

This may sound weird, but I think the real issue has something to do with Kubernetes. This config works (populates the admin page without the error) on docker/EC2 but not on EKS.

from jupyterhub.auth import Authenticator

class CustomAuthenticator(Authenticator):

    async def authenticate(self, handler, data):
        username = data['username']
        pwd = data['password']

        return {
            'name': username,
        }

c = get_config()  # noqa

c.JupyterHub.authenticator_class = CustomAuthenticator
c.CustomAuthenticator.admin_users = {'gramesh'}

The deployment spec

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jupyterhub
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: jupyterhub
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jupyterhub
    spec:
      containers:
      - name: hub
        image: company_registry/image

The version of EKS is 1.19.13

This could also be due to the load balancer that’s fronting Jupyterhub (AWS ALB). Something to do with header/cookies? I’m not sure how to debug this.

6

Load balancer mismanaging cookies or some such does sound possible. You might need to inject some extra debugging to dump the headers of each request and compare the successful and rejected requests. There’s code in jupyterhub/log.py that’s called on every request that might help.