Bug in 2.0.1? Pending spawn_pending after login due to missing read:servers scope

Similar looking to #2815 but running Jupyterhub==2.0.1 with oauthenticator==14.2.0 and a genricoauthenticator.
Error seems fairly clear:
[W 2021-12-29 22:07:41.380 JupyterHub scopes:496] Not authorizing access to /jupyter/hub/api/users/[user]/server/progress. Requires any of [read:servers], not derived from scopes []
but the basic user role does not have that scope.
Can it be added to the user role or is there a workaround other than having users click home?

Thanks!

1 Like

Do you have a custom user role defined? The default user role does have this permission for their own servers.

The error message suggests that the request is made by a user who does not have the user role, or any role, because they have no permissions.

Do you have any roles defined, and are you using authorized_users config?

No custom user roles and am not setting allowed_users. So from my understanding, any user authenticated from my genericoauth should be allowed.

For now I have resolved by reverting to 1.4.2

Is it possible to share your config with secrets removed? And logs from jupyterhub --debug including the startup logs where it notes things about initial users?

The issues of users not having or losing their user role membership that should have been fixed in 2.0.1 seems like they would be the cause, but it’s possible there are more not yet covered.

Hi, Unfortunately I ran into the same error. I can not see any kind of spawn progress. Stopping my notebook server is also not possible.

Config:

# Copyright (c) Jupyter Development Team
# Distributed under the terms of the Modified BSD License.

# Configuration file for JupyterHub
import os
import sys
from oauthenticator.generic import GenericOAuthenticator

c = get_config()

c.JupyterHub.log_level = 'DEBUG'

c.NotebookApp.allow_root=True

c.ConfigurableHTTPProxy.auth_token = '<removed>'
c.JupyterHub.authenticate_prometheus = False
c.JupyterHub.cookie_secret = bytes.fromhex('<removed>')

c.Spawner.start_timeout = 60
c.Spawner.http_timeout = 60

c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'

c.DockerSpawner.image = '<removed>'

c.Spawner.default_url = '/lab'
c.DockerSpawner.cmd = "start-notebook.sh"

c.Spawner.mem_limit = '10G'
network_name = os.environ['DOCKER_NETWORK_NAME']
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = { 'network_mode': network_name, 'tmpfs': ['/tmp:rw,size=500M'] }

c.DockerSpawner.extra_host_config.update({ 'oom_score_adj': 500, 'mem_reservation': '4G', 'cpu_shares': 512 })

c.DockerSpawner.remove_containers = True
c.DockerSpawner.debug = True
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = 8080

c.JupyterHub.port = 1080

# Custom Authentication

c.JupyterHub.authenticator_class = GenericOAuthenticator
c.GenericOAuthenticator.client_id = '<removed>'
c.GenericOAuthenticator.client_secret = '<removed>'
c.GenericOAuthenticator.authorize_url = 'https://<removed>/protocol/openid-connect/auth'
c.GenericOAuthenticator.token_url = 'https://<removed>/protocol/openid-connect/token'
c.GenericOAuthenticator.userdata_url = 'https://<removed>/protocol/openid-connect/userinfo'
c.GenericOAuthenticator.oauth_callback_url = 'https://<removed>/hub/oauth_callback'
c.GenericOAuthenticator.userdata_params = {'state': 'state'}
c.GenericOAuthenticator.username_key = 'preferred_username'
c.GenericOAuthenticator.login_service = 'EXAMPLE'
c.GenericOAuthenticator.scope = ['openid', 'profile', 'userservices']
c.GenericOAuthenticator.claim_groups_key = 'user_services'
c.GenericOAuthenticator.allowed_groups = ['jupyterAccess']

c.JupyterHub.db_url = os.path.join('sqlite:///', '/data', 'jupyterhub.sqlite')
c.JupyterHub.cookie_secret_file = os.path.join('/data', 'jupyterhub_cookie_secret')

The following logs are from the initial jupyterhub starting, spawning a single user notebook and the trying to stop the notebook server via control panel:

Sorry for the delay.

# JupyterHub configuration

import os
import sys
import subprocess
import pwd
from oauthenticator.generic import GenericOAuthenticator
import pyodbc

# Generic
c.JupyterHub.admin_access = True
c.Spawner.default_url = '/lab'

c.JupyterHub.authenticator_class = GenericOAuthenticator

c.GenericOAuthenticator.update({
    'client_id': os.getenv('OAUTH_CLIENT_ID'),
    'client_secret': os.getenv('OAUTH_CLIENT_SECRET'),
    'login_service': os.getenv('OAUTH2_LOGIN_SERVICE'),
    'username_key': os.getenv('OAUTH2_USERNAME_KEY'),
    'userdata_method': 'GET',
    'scope': [
        'openid',
        'profile',
        'email',
        'offline_access',
    ]
})

c.Authenticator.allowed_users = set()

c.Authenticator.admin_users = {}


def pre_spawn_hook(spawner):
    # https://github.com/jupyterhub/jupyterhub/issues/2948#issuecomment-606243514
    username = spawner.user.name
    try:
        pwd.getpwnam(username)
    except KeyError:
        subprocess.check_call(['useradd', '-ms', '/bin/bash', username])

c.Spawner.pre_spawn_hook = pre_spawn_hook

# Docker spawner
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = os.environ['DOCKER_JUPYTER_CONTAINER']
c.DockerSpawner.network_name = os.environ['DOCKER_NETWORK_NAME']
# See https://github.com/jupyterhub/dockerspawner/blob/master/examples/oauth/jupyterhub_config.py
c.JupyterHub.hub_ip = os.environ['HUB_IP']
c.JupyterHub.bind_url = 'http://0.0.0.0:8000/jupyter/'

# user data persistence
# https://jupyterhub-dockerspawner.readthedocs.io/en/latest/data-persistence.html
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR')
workingdir = os.environ.get('WORKINGDIR')
c.DockerSpawner.notebook_dir = notebook_dir

c.DockerSpawner.volumes = {
    'jupyterhub-user-{username}': notebook_dir,
}

c.JupyterHub.load_roles = [
    {"name": "jupyterhub-idle-culler-role",
     "scopes": [
         "list:users",
         "read:users:activity",
         "read:servers",
         "delete:servers",
         # "admin:users", # if using --cull-users
     ],
     # assignment of role's permissions to:
     "services": ["jupyterhub-idle-culler-service"],
    }
]

c.JupyterHub.services = [
    {
        "name": "jupyterhub-idle-culler-service",
        "command": [
            sys.executable,
            "-m", "jupyterhub_idle_culler",
            "--timeout=3600",
        ],
        # "admin": True,  # may be able to remove once on jupyterhub 2.0+
    }
]

Log:

[I 2022-01-22 00:06:41.520 JupyterHub log:189] 302 GET /jupyter/hub/ -> /jupyter/hub/login?next=%2Fjupyter%2Fhub%2F (@10.5.8.175) 1.14ms
[I 2022-01-22 00:06:41.589 JupyterHub log:189] 200 GET /jupyter/hub/login?next=%2Fjupyter%2Fhub%2F (@10.5.8.175) 19.09ms
[I 2022-01-22 00:06:43.363 JupyterHub oauth2:111] OAuth redirect: 'https://mysite/jupyter/hub/oauth_callback'
[I 2022-01-22 00:06:43.364 JupyterHub log:189] 302 GET /jupyter/hub/oauth_login?next=%2Fjupyter%2Fhub%2F -> https://co.okta.com/oauth2/v1/bla (@10.5.8.175) 1.37ms
[I 2022-01-22 00:06:43.969 JupyterHub base:797] User logged in: email@co.com
[I 2022-01-22 00:06:43.970 JupyterHub log:189] 302 GET /jupyter/hub/oauth_callback?code=[secret]&state=[secret] -> /jupyter/hub/ (@10.5.8.175) 263.28ms
[I 2022-01-22 00:06:44.016 JupyterHub log:189] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn (email@co.com@10.5.8.175) 9.44ms
[I 2022-01-22 00:06:44.066 JupyterHub roles:477] Adding role server to token: <APIToken('c53f...', user='email@co.com', client_id='jupyterhub')>
[I 2022-01-22 00:06:44.074 JupyterHub provider:607] Creating oauth client jupyterhub-user-email%40co.com
[I 2022-01-22 00:06:44.158 JupyterHub dockerspawner:988] Container 'jupyter-email-40co-2ecom' is gone
[I 2022-01-22 00:06:44.195 JupyterHub dockerspawner:1272] Created container jupyter-email-40co-2ecom (id: 5f2a926) from image jupyterlab_img
[I 2022-01-22 00:06:44.195 JupyterHub dockerspawner:1296] Starting container jupyter-email-40co-2ecom (id: 5f2a926)
[I 2022-01-22 00:06:45.065 JupyterHub log:189] 302 GET /jupyter/hub/spawn -> /jupyter/hub/spawn-pending/email@co.com (email@co.com@10.5.8.175) 1003.87ms
[I 2022-01-22 00:06:45.103 JupyterHub pages:400] email@co.com is pending spawn
[I 2022-01-22 00:06:45.106 JupyterHub log:189] 200 GET /jupyter/hub/spawn-pending/email@co.com (email@co.com@10.5.8.175) 5.72ms
[W 2022-01-22 00:06:45.241 JupyterHub base:89] Blocking Cross Origin API request.  Referer: https://mysite/jupyter/hub/spawn-pending/email@co.com, Host: mysite, Host URL: http://mysite/jupyter/hub/
[W 2022-01-22 00:06:45.241 JupyterHub scopes:496] Not authorizing access to /jupyter/hub/api/users/email@co.com/server/progress. Requires any of [read:servers], not derived from scopes []
[W 2022-01-22 00:06:45.242 JupyterHub web:1787] 403 GET /jupyter/hub/api/users/email@co.com/server/progress (10.5.8.175): Action is not authorized with current scopes; requires any of [read:servers]
[W 2022-01-22 00:06:45.242 JupyterHub log:189] 403 GET /jupyter/hub/api/users/email@co.com/server/progress (@10.5.8.175) 2.29ms
[I 2022-01-22 00:06:45.645 JupyterHub log:189] 200 GET /jupyter/hub/api (@172.27.0.3) 0.60ms
[I 2022-01-22 00:06:45.664 JupyterHub log:189] 200 POST /jupyter/hub/api/users/email@co.com/activity (email@co.com@172.27.0.3) 12.77ms
[I 2022-01-22 00:06:45.902 JupyterHub base:944] User email@co.com took 1.839 seconds to start
[I 2022-01-22 00:06:45.902 JupyterHub proxy:286] Adding user email@co.com to proxy /jupyter/user/email@co.com/ => http://172.27.0.3:8888
00:06:45.903 [ConfigProxy] info: Adding route /jupyter/user/email@co.com -> http://172.27.0.3:8888
00:06:45.903 [ConfigProxy] info: Route added /jupyter/user/email@co.com -> http://172.27.0.3:8888
00:06:45.903 [ConfigProxy] info: 201 POST /api/routes/jupyter/user/email@co.com

Hi together,

I’m seeing the same issue. From my point of view this is caused by the CORS change.